An important source of information for virtual validation is the datasets generated from multi-center cancer research projects such as The Cancer Genome Atlas project (TCGA). These data enable investigation of genetic and epigenetic changes responsible for cancer onset and progression, response to cancer therapies, and discovery of the molecular profiles of various cancers. However, these analyses often require bulk download of data and substantial bioinformatics expertise, which can be intimidating for investigators. Here, we report on the development of a new resource available to scientists: a data base called Glioblastoma Bio Discovery Portal (GBM-BioDP). GBM-BioDP is a free web-accessible resource that hosts a subset of the glioblastoma TCGA data and enables an intuitive query and interactive display of the resultant data. This resource provides visualization tools for the exploration of gene, miRNA, and protein expression, differential expression within the subtypes of GBM, and potential associations with clinical outcome, which are useful for virtual biological validation. The tool may also enable generation of hypotheses on how therapies impact GBM molecular profiles, which can help in personalization of treatment for optimal outcome. The resource can be accessed freely at  The Cancer Genome Atlas (TCGA) project The expanding quantity of data and proliferation of results from computational analyses create the challenge of making the experimental data and accumulated knowledge accessible to a wide range of researchers. Bulk downloads and re-processing of data is unappealing in most situations and only an option to bioinformatics experts in search of new ways to analyze the data. Most researchers and bench scientists need easily-accessible data browsers and visualization tools that present abstracted, intuitive, and integrated views of the data, and that enable validation of biological insights, or generation of new hypotheses. To this end we present the Glioblastoma BioDiscovery Portal (GBM-BioDP) – an intuitive, integrative, web-accessible data portal that contains a subset of the TCGA glioblastoma (GBM) data, and incorporates knowledge from computational analyses and predictions.GBM is the most common and aggressive malignant primary brain tumor. Patients with GBM have a poor prognosis and usually the standard first-line of treatment is surgery, followed by radiation therapy or combined radiation and chemotherapy (such as temozolomide). Unfortunately, these treatments are rarely curative and the vast majority of tumors recur locally within the brain, with a median survival of 15 months classical, mesenchymal, proneural, and neural  GBM is extremely heterogeneous and arises from dysregulation of a number of important biological pathways, which makes understanding the responsible molecular mechanisms difficult. Nevertheless, computational analyses of multidimensional TCGA data are improving our understanding of the disease. Four molecular subtype profiles of the disease have been identified by computational tools: GBM-BioDP has two main uses: biological validation, and new hypothesis generation. First, it can answer gene or miRNA specific questions, such as: Is a gene of interest changed in any of the molecular subtypes of GBM? Is its expression level associated with the clinical outcome of GBM patients? Do the expression and target prediction data support a regulatory relationship between given genes and miRNAs? Many publications have identified specific genes or miRNAs whose dysregulation has been associated with GBM and whose expression levels are associated with better or worse prognosis. For bench scientists our resource makes it possible to explore the stipulated relationships visually. The second way the resource can be utilized is as a sand box for hypothesis generation. Cancer therapies impact the cancer molecular profiles. User provided expression data can be compared against the known GBM subtypes and classified using several prediction models. For treated conditions the user can inspect whether treatment modifies the classification, thus making the cancer more or less aggressive.Related tools for analyzing, integrating, and visualizing cancer genomic data have been developed alongside experimental data generation. A recent overview of cancer genomic visualization tools is given by Schroeder et al. Two of the most commonly used freely available, web-accessible tools are the UCSC Cancer Genomics Browser Some of the features that make our tool unique are: multiple ways of stratifying samples when evaluating clinical impact  prognostic index), aggregation of expression data from different platforms, incorporation of miRNA expression as well as regulator-target relationship predictions, and classification of user-input expression data. We are also working towards integration of other dimensions, such as copy number, and DNA methylation.In the reminder of this article we describe the tool construction, core features, and use cases.https://tcga-data.nci.nih.gov/) as described in The Cancer Genome Atlas research Network Experimental and clinical data were downloaded from the TCGA data portal  data from three platforms HT_HG-U133A (488 patient samples×12042 features), HuEx-1_0-st-v2 (437 patient samples×18631 features), AgilentG4502A_07_1/2 (101+396 patient sample×17813 features). The data from the three platforms were aggregated following Protein expression data include normalized (level 2) Reverse Phase Protein Array data (217 patient samples×171 features). 45 of the featured antibodies interrogate the phosphorylation status of the proteins.miRNA expression consist of normalized (level 3) data from H-miRNA_8×15 K platform (436 patient samples×534 features).Clinical data include partial clinical information on 564 patients.The experimental (level 2 and 3) data were already pre-processed as part of the TCGA. For protein expression (level 2), preprocessing include: centering, z-score transformation with corresponding p-values, and summarizations such as percentage of under/over-expressed features.Data from several types of computational analysis were incorporated in the knowledge base.classical, mesenchymal, proneural, and neural . Data from Verhaak et al. Verhaak et al. We also incorporate computational predictions of miRNA targets from MicroCosm. Predicted targets can be used in conjunction with expression data to examine regulator-target relationships between miRNAs and genes or proteins.http://cran.r-project.org) The visualizer was constructed as a tiered application. The high-level architecture is shown in Statistical analyses were performed in R and the results are incorporated into visual displays. Four types of analyses are made: descriptive, prognostic, correlation, and predictive. These analyses are performed on the patients stratified by molecular subtypes as detailed earlier, as well as on the total cohort selected. Examples of these analyses are described in the online tutorial provided in GBM-BioDP website.Descriptive analyses: Box and whiskers plots permit to graphically represent descriptive statistics of a continuous variable (e.g. gene). Here, box and whiskers plots are presented to visually compare distributions of a gene among the different molecular subtypes. When there is more than one molecular subtype, Welch’s test is used to evaluate the difference of gene’s expression among the subtypes.Prognostic analyses: Prognostic analyses were performed at the targeted single molecule  level or multiple selection criteria.A. Targeted prognostic analyses for chosen criteria include several statistical tests. These tests were conducted on GBM subtypes and total cohort with data previously converted to a common scale with a suitable z-score normalization. The prognostic impact of each expression measure is evaluated by means of univariate Cox proportional hazards model. Results are displayed according to the molecular subtype, and total cohort. Kaplan-Meier curves were then performed with expression values dichotomized according to the median value. Cox results corresponding to dichotomized values were displayed on the curve. An option is provided to group patients based on quartile selection of the expression measure.B. Multiple selection criteria: For a list of expression measures, two types of analyses were performed.Cluster based prognostic analyses, in which patients were stratified by hierarchical clustering based on the selected list of molecules. Using silhouette width index, optimum clusters were selected. A log rank statistic was performed using Kaplan Meier curves to test the differences between the cluster groups. These analyses were performed using “NbClust” package in R.1x1+β2x2+…+βpxpβ where ix is the expression value and the Iβ can be obtained from the Cox fitting. Each Iβ can be interpreted as a risk coefficient. The fitting is performed in R using the “survival” package. The risk groups were dichotomized by ordered PI by median value  leaving equal number of samples in each group, or by quartile selection. For the resulting two groups, a log-rank test is performed.Prognostic index estimation. The prognostic index (PI), also known as the risk score, is commonly used to generate risk groups. The PI is known as the linear component of the Cox model, PI = Correlation analyses: A pairwise Pearson correlation analysis was performed for the user selected list of molecules. The results were displayed as heat map using hierarchical cluster analysis using average linkage distance metric.Predictive analyses: To classify new patients or samples into the known molecular subtypes of GBM, two prediction methods were implemented. These analyses were performed using the Bioconductor packages “pamr” and “randomForest”.A. Prediction analysis of microarrays: Briefly, PAM B. Random Forest: The Random Forest The interface is organized into modules. Currently the three main modules are centered on the concept of Genes, miRNAs, and (prediction) Models.Genes and miRNAs modules provide analogous features centered on the molecule  or molecule product (protein) in question.Expression profiles can be visualized for single or multiple molecules. For single molecules, the expression profile visualization includes a histogram of the distribution of the zscore-expression values , as wellExpression profiles of multiple molecules can be inspected by constructing heatmaps that cluster according to similarity.In conjunction with expression profiles, the user can perform survival analyses that consider the impact of a molecule’s expression levels on the length patient survival. When patients are stratified according to their molecular subtype  survival analysis can further stratify samples according to the expression levels of the molecule in question. Kaplan-Meier curves, or Cox proportional hazards models can be constructed that compare the impact of high or low expression values on the length of survival . For CoxFor multiple molecules, two types of survival analysis can be performed. First, the samples are clustered by similarity of expression profiles of the molecules in question, and Kaplan Meier analysis displays any differences in survival based on cluster membership. Second, a PI is generated for the user defined selection . A Cox model is constructed with this prognostic index as covariate, and optional covariates for Age and MGMT promoter methylation status. The effect of different levels of the prognostic index is displayed for the full cohort, and for each GBM subclass.Regulator-target relationships among miRNA and gene products can be explored by constructing expression correlation heatmaps between miRNAs and gene products that are computationally predicted to be in such a relationship based on sequence similarity Models module enables classification of user supplied gene or miRNA expression data into the four molecular subtypes of GBM by supervised classification methods. Currently we provide options for two classification methods (PAM and Random Forest). The results include predicted classification and associated probabilities, plots for visualizing the profiles of the top classifiers, and heatmaps of correlation between model and user input data.Here we give an example of biological validation from miRNAs module, demonstrating how the profiles of genes and miRNAs recently identified in the literature as having an important mechanistic and prognostic role in GBM can be inspected using GBM-BioDP.Micro RNAs have a key regulatory role, directly and indirectly impacting the expression level of genes. In particular, mir-34a has been recently studied in connection with its regulatory role in GBM. Genovese et al. A more recent study by Sasaki et al. In light of Sasaki’s study, we inspected whether lower levels of mir-34a are in general associated with better survival. To inspect the relationship between mir-34a, and p53 protein expression levels we constructed an expression correlation heatmap between mir-34a, and its predicted targets in Microcosm. Note that p53 is not one of the top predicted targets of mir-34a (based on sequence similarity), therefore we included in the heatmap mir-152, which is one of the strongest predicted regulators of p53 (see predicted regulators of p53 from the Genes module). The correlation heatmap  points tGBM-BioDP can be used to predict the molecular subtypes of new experimental, or clinical samples into known molecular subtypes, which can aid in improving understanding of GBM neurogenesis, constructing experimental models, and tailoring therapies. We illustrate the use of the tool by looking at the classification of grade IV gliomas (GSE4271) that were used by Phillips et al. pronerual, proliferative, and mesenchymal), which were associated with key stages in neurogenesis and implicated signaling pathways critical in regulation of forebrain neurogenesis and tumor aggressiveness. The authors also demonstrated a frequent pattern of glioma progression into the mesenchymal phenotype, a state associated with robust angiogenesis. More recently, Verhaak et al. neural and classical; see Phillips et al. We tested the molecular class prediction based on the expression profiles of an independent set of patients with grade IV gliomas (GSE4271). The data included patients from primary or recurrent GBM presenting with or without necrosis see . We recaOur model correctly predicted proneural subtypes with an error rate of 0.08. The median age of patients in this group is 31 years, supporting earlier findings that patients classified as proneural are of younger age group. The majority of mesenchymal samples were also predicted into the correct class . However, wide heterogeneity was observed for the proliferative class. Recall that the proliferative class designated by Phillips et al. was further subclassified into neural and classical by Verhaak et al. . In our Advances in biotechnology have brought about accumulation of large scale cancer genomic data, in turn introducing the challenge of making the data available to a wide array of researches, scientists, and clinicians. While several impressive resources that offer visualization options for various omic data exist, no such tool can be used in all possible scenarios. We present a new resource for visualizing GBM TCGA data, which aims at providing streamlined visualizations of common scenarios when investigating molecular profiles of glioblastoma, and how these profiles are associated with clinical outcomes.The resource offers integrated views of several dimensions of TCGA data, such as mRNA, protein, and miRNA expression, exploration of target-regulator relationships, as well as provides predictive models for user input data. Molecular profiles can be explored based on a number of sample stratification options . Conversely, clinical outcomes can be explored for samples exhibiting different molecular profiles. The resource can be used for biological validation, as well as generation of new hypothesis.We are currently developing options for integrating views from other TCGA omic data including copy number, SNP, and DNA methylation. Another active area of development is integration at the functional and biological pathway enrichment level.


Multiple methods have been proposed to estimate pathway activities from expression profiles, and yet, there is not enough information available about the performance of those methods. This makes selection of a suitable tool for pathway analysis difficult. Although methods based on simple gene lists have remained the most common approach, various methods that also consider pathway structure have emerged. To provide practical insight about the performance of both list-based and structure-based methods, we tested six different approaches to estimate pathway activities in two different case study settings of different characteristics. The first case study setting involved six renal cell cancer data sets, and the differences between expression profiles of case and control samples were relatively big. The second case study setting involved four type 1 diabetes data sets, and the profiles of case and control samples were more similar to each other. In general, there were marked differences in the outcomes of the different pathway tools even with the same input data. In the cancer studies, the results of a tested method were typically consistent across the different data sets, yet different between the methods. In the more challenging diabetes studies, almost all the tested methods detected as significant only few pathways if any. In the past years, pathway analysis has become a common operation in functional genomics studies, as investigating single gene activities alone has turned out not to be sufficient. Multiple tools for pathway analysis have been proposed, making the selection of a suitable tool difficult. Most of the tools simply consider the pathways as unstructured gene sets, and define the pathway activity as enrichment of the pathway genes among the top detections . There aG, a pathway that consists of p genes m samples including a subset of case samples CA and a subset of control samples RA. Genes are indexed with i and samples with j. The term DE genes refers to differentially expressed genes.In this section, we shortly describe the mathematical basis of each of the six tested methods. Some basic information about features and versions of the methods applied in this study is presented in G is defined asNDEP is the probability that the pathway includes at least the observed number of DE genes when the null hypothesis is true, and PBP is the probability that the pathway has at least as high total perturbation as observed (assuming again null hypothesis). The null hypothesis for NDEP is that all DE genes are distributed randomly in a list of measured genes, and for PBP that the pathway DE genes take random places in the pathway. Details about the calculation of NDEP and PBP are provided in the original publication [G is calculated as a sum of the accumulated perturbations of the genes in the pathway:i (log fold-change ratio). The term PF(i) corresponds to the perturbation of gene i, including both measured perturbation and perturbation inherited from its parent nodes and is defined asparent(i) refers to parent nodes of gene i, and k. Coefficient kiβ tells the type of interaction between parent k and child i (1 for activation and −1 for inhibition).In SPIA, the score for pathway lication . Total pG is defined asThe centrality-based pathway enrichment tool CePa includes multiple different ways to consider pathway structure . In thiscentrality(i) can have different definitions, and it is up to the user to choose which one to use. It can be, for example, the length of the longest shortest path to leaf node, the length of the longest shortest path to root node, the number of child nodes or the number of parent nodes. To avoid favoring pathways with certain structure (e.g. chain-like pathways), multiple centrality criteria are used.The term je of sample j consists of real signal and noise and can be defined asjγ is a latent variable representing the individual effect. The matrix Λ is a weighted influence matrix that contains the information about the relations between the measured genes. The NetGSA test statistic for pathway G is then defined asGb indicates which genes belong to pathway G and CX and RX are matrices including vectors je as columns, where j belongs to case samples CA and control samples RA, respectively. Testing the null hypothesis The mathematical model behind NetGSA is rather complex, so we describe here only its general concept. More detailed explanation is available in the original papers , 17. TheThe DAVID tool is based on modified Fisher’s exact test. In the basic Fisher’s exact test, genes are divided into two groups based on two criteria: whether a gene is DE, and whether it belongs to a specific pathway. Then the probability of having a given number of DE genes in a pathway is calculated using hypergeometric distribution. DAVID uses Fisher’s exact test with jackknifing , 19. Than genes in the data. In a typical case, the ranking of a gene i is done according to differential expression d(i) between two groups of samples, for example, healthy and sick. After the ranked list of genes has been formed, an output value GSEAS can be calculated for each pathway (gene set) G. The score a cumulative sum, which can be formulated asG up to a given rank r, and G. The term d(i) is an estimate of differential expression of gene i, r(i) is the rank of gene i and u can have different values. The most common choices for u are u = 0 and u = 1. The term p is the number of genes in pathway gene set G. The significance P-value of The first step in GSEA is to form a decreasing ranked list, which consists of all the PathifierS for each sample G. When analyzing pathway G, only gene expression measurements of genes belonging into G are considered. Now all the samples can be reduced to vectors of length p, where p is number of genes in pathway G. The score is based on nonlinear principal curve [js and pathway G is the distance between the projection of the reduced sample x and y along the curve z. Now Pathifier score pc is the principal curve and function projec returns the projection of a particular sample to the principal curve Unlike other methods considered here, the Pathifier tool calculates a score al curve  generateBecause SPIA, CePa and DAVID need a list of DE genes as input, we used two tools to find the DE genes: Limma (version 3.22.4)  and ROTSAll data sets were tested as unscaled measurements and as base-two logarithm–scaled measurements. Before logarithm transformation, a constant one was added to all measurements to prevent negative values without causing big changes in the general range of measured values. If the authors of a method have advised that the method should be used with scaled or unscaled input data, this advice was respected. Otherwise both preprocessing approaches were considered. GSEA takes as input unscaled data, and Pathifier and NetGSA take as input logarithm–scaled data. For the methods that use as input lists of DE genes , there were no such recommendations.P-values, they were converted to FDR values using the Benjamini–Hochberg method. With CePa, a pathway is considered as significant if it has FDR value < 0.05 according to at least one of six ready-made ways to define the centralities.In the comparisons, the cutoff value for significant pathways was defined so that false discovery rate (FDR) was <0.05. If a method returns only To have comparable results from different methods, the same pathways should be analyzed. We used KEGG database  as a patl describes the total number of tested data sets and it is six for ccRCC and four for T1D, and β tells how many pathways a given method found from exactly h data sets. Function α tells how many pathways the given method found in artificial data sets on average . Constant 1 is added to the denominator to avoid dividing by 0. The scaling term T is the theoretical maximum score, which depends on the number of tested pathways and used data sets. With 86 pathways and six data sets (ccRCC tests), T gets a value of score to make ccRCC and T1D test scores comparable. Because in practice, the scores are much lower than the theoretical maximum, a coefficient 1000 is included to keep the general level of scores in a readable level. The score is calculated separately for each method with scaled and original data and with Limma and ROTS, if applicable.To evaluate the different pathway methods, multiple data sets from two different conditions were considered, including six data sets on clear cell renal cell carcinoma (ccRCC) and four data sets on type 1 diabetes (T1D). A method is considered as reliable if it returns consistent results from similar study settings. Results are expected to vary to some extent because of individual differences and differences in study designs such as measurement protocol, number of samples and the type of control samples (e.g. paired or not). To compare different methods, we developed a scoring method based on weighted number of pathways found from multiple data sets scaled by the average number of false positives. The score is defined asData from two different diseases representing different characteristics were used to test and compare the pathway methods. In ccRCC data sets, individuals are heterogeneous, but changes between sick and healthy samples are relatively big. In a more difficult case of T1D, the data are heterogeneous and differences between the sample groups are relatively small. This causes ccRCC data sets to include typically thousands of DE genes, whereas <20 genes appear as DE in T1D data sets.Six ccRCC data sets were downloaded from the GEO database  and theyIf a ccRCC data set included samples other than ccRCC or healthy controls, they were left out from analysis. With T1D data sets, the selection of samples to be used was more complicated because the T1D data sets often included time series measurements or matched control samples. From T1D data set GSE9006, we compared the healthy control samples with samples taken from T1D patients 4 months after T1D diagnosis. From data sets GSE30211 and TABM-666, we compared newly diagnosed T1D samples at the time of diagnosis and strictly matched control samples, matched by age, gender, date of birth and genetic risk. In data set GSE51058, we compared individuals who developed T1D with those who did not, using measurements at seroconversion to autoantibody positivity.The microarray data sets included original probes/probesets instead of genes and they needed to be transformed before pathway analysis. In case one probe/probeset referred to multiple genes, it was removed from the analysis. In case multiple probes/probesets referred to one gene, just one of them was selected based on the highest variance among all the samples.Besides the 10 real comparisons (six in ccRCC and four in T1D), artificial data sets were also tested. The purpose of these artificial data sets was to find out if the methods favor some type of pathways or find plenty of noise pathways in general. The artificial data sets were generated by picking all controls from one T1D and one ccRCC data set and randomly dividing them into artificial case and control groups. Five unscaled and five logarithm-scaled data sets were generated from the ccRCC data set, and five unscaled and five scaled data sets were generated from the T1D data set.In this section, we describe our results for the tested pathway methods in the 10 ccRCC or T1D data sets, as well as additional 10 artificial data sets where no significant pathways are expected. To ensure comparability between the computational methods, we focus on pathways supported by all of the tools. In the subsection ‘Comparison between ROTS and Limma’, we compare results from Limma and ROTS, otherwise we concentrate on results obtained when the list of DE genes was determined by ROTS. A general overview of the consistency of the results is presented in a heatmap format in In general, the results from the different methods were not similar in the six tested different ccRCC data sets A. SignifSPIA found rather many significant pathways from the ccRCC data sets. Although there were several pathways found from only one data set, in general, results from SPIA were consistent. From scaled ccRCC data sets, 11 pathways were detected consistently from five or six of the six ccRCC data sets. Results from scaled and unscaled data sets were not identical, but the general trends were mostly similar. From artificial data sets, SPIA detected only one false-positive significant pathway from one data set.Scaling had rather big effects on the CePa results. Without scaling, the results were not consistent between different data sets. With logarithm-scaled data sets, two pathways were marked as significant for all data sets. These pathways were PPAR signaling pathway (KEGG accession hsa03320) and leukocyte transendothelial migration (KEGG accession hsa04670). Findings from artificial data sets were few and not consistent.With NetGSA, one data set (GSE15641) had almost all pathways detected as significant; yet there were not any significant findings from the other data sets. With the other methods, the contrast between GSE15641 and other data sets was not apparent. This indicates that with NetGSA, cutoff values should be carefully adjusted for the data set under consideration. From artificial data sets, NetGSA did not find any pathways significant.GSEA and DAVID made few significant findings in our comparisons and pathway selection. None of the pathways was found in more than four data sets by DAVID and in more than three data sets by GSEA. GSEA was tested only with unscaled data sets because of the author’s recommendations, but DAVID was tested with both scaled and unscaled data sets. From unscaled data sets, DAVID found more significant pathways than from scaled data sets. Both DAVID and GSEA detected few, if any, pathways significant from artificial data sets.Pathifier results with different data sets were consistent, but not informative. Data set GSE14994 had one pathway not detected as significant, otherwise all pathways from all the data sets were significant. Based on the author’s recommendation, only scaled data sets were tested. There were also more findings from artificial data sets compared with the other methods. There is one interesting side about Pathifier results with artificial data. Results from the artificial data sets generated based on controls from T1D data set are greatly different from those from ccRCC-based artificial data. From the ccRCC-based data, few noise pathways were detected, whereas from the T1D-based artificial data sets, plenty of significant pathways were found, which are considered as false positives. Such phenomena did not appear with the other tested pathway methods.Supplementary Tables S1 and S2 respectively.To sum up, in our tests with ccRCC data sets, two methods, SPIA and CePa, had the best balance between consistency and the amount of results. For consistent CePa results, scaling of the data was necessary. Other four methods detected typically few or many pathways significant with the test design used here. Complete tables of significant pathways detected from artificial and ccRCC data sets by different methods and preprocessing procedures are available in In T1D, the differences between individuals were relatively big compared with disease-related differences, which makes the T1D data more challenging to analyze. In line with this, all the methods made fewer findings from the T1D data sets than from the ccRCC data sets B. EspeciThe most interesting T1D results were from SPIA when using logarithm-scaled data sets. Three pathways  were found significant from half of the data sets, but not from the artificial data sets.CePa found some pathways significant, but those findings were few and not consistent between data sets. Also from artificial data sets, CePa detected only few pathways significant. Findings from T1D data sets and artificial data sets were not similar to each other.Two of the methods not using pathway topology (DAVID and GSEA) and one topology-using method (NetGSA) did not find anything from the challenging T1D data sets. These methods had also few, if any, false-positive findings from artificial data sets.Pathifier again showed more findings than the other methods. Pathifier was not able to analyze the data set TABM-666 because of too few, three, control samples, and from GSE51058 there were otherwise few findings. From the two remaining data sets, there were plenty of significant pathways. Several pathways were repeatedly found also in the artificial data sets.Supplementary Table S3.In general, T1D data sets turned out to be challenging to all the methods. Five of the six tested methods detected few, if any, significant pathways from T1D data sets. The one remaining method, Pathifier, detected around the same amount of significant pathways from T1D data sets and artificial data sets. This makes it difficult to estimate whether Pathifier results are real findings or false positives. A complete table of significant pathways detected from T1D data sets by different methods and preprocessing procedures is available in Supplementary Table S4.To quantify the consistency of the methods across the six ccRCC data sets or the four T1D data sets, we used the scores defined as in (7). The aim was not to favor either methods with plenty of findings but many false positives or methods with no false positives but only few findings. The scores of the methods with all combinations of scaled and unscaled data sets and ROTS and Limma DE gene detection are presented in The highest scores of the methods were typically obtained using ROTS in logarithm-scaled data. If Limma is used to detect DE genes, the results with scaled and unscaled data sets were, in general, closer to each other than the corresponding ROTS-based results. With ROTS, the results were better with scaled input data. The case where preprocessing procedures  had most impact on results was ccRCC tests with CePa.The scores were calculated also for the results from the artificial data sets by dividing the artificial results randomly into two groups of real and artificial results. The scores from artificial results were zero for most of the methods and preprocessing approaches, and they did not exceed 0.005 in any of the cases.In the comparisons, we concentrated on the mathematical algorithms behind the methods and minimized the effect of other properties, such as the selected data base. However, the other ignored features can have a marked effect on the choice of a method or the final results. For instance, for an average user, the ease of use and clear documentation and instructions play a major role when selecting the method.SPIA was easy to use and the results included information not only about significance of pathway, but also direction of perturbance and size of the tested pathways.Features of CePa relevant for user are freedom to define which nodes of graph are weighted most. CePa also searches for pathways from multiple data bases and is easy and fast to use. A negative side of the automatic pathway search is that the newest pathways at least in KEGG are not available.NetGSA results could probably be improved by putting more effort into forming input files. One of the input files is a matrix that includes all known gene interactions. For this study, we used those relations that appear in any pathway, but this could be improved by including also other relations found from literature or other sources. During this study, NetGSA implementation was still under development.The strength of DAVID is that it is easy and fast to use and does not require heavy preprocessing. It needs only a list of DE genes and even the gene ID type can be any of the common ones. Also, it uses large selection of pathways from different data bases and includes plenty of other functions than pathway analysis as well, such as clustering and gene ID converting.Like DAVID, GSEA also includes a detailed user manual and it is fast and easy to use. In addition, it can investigate, for example, time series data or data with multiple groups.Only Pathifier of the six tested methods returns sample-level results. Converting those results to group level with Limma and ROTS and cutoff limit FDR <0.05 provided typically almost all or none of the tested pathways to be significant. This indicates that the cutoff value should be more carefully chosen or that the results should be considered only in sample level.As expected, all the methods tested found more significant pathways from ccRCC data sets than from T1D data sets. Notably, the structure-based methods  found more results than the non–structure-based ones. This is also a natural outcome because the methods using pathway structure have more input information than the methods based on simple gene lists.Methods using pathway structure loosely (SPIA and CePa) performed better in our tests than the method using detailed pathway structure (NetGSA). It seems that more complex methods, NetGSA and Pathifier, are sensitive to the set cutoff value because they tend to find either all or none of the pathways as significant in a given data set. In the challenging T1D data sets, none of the tested methods found consistent results. Taken together, our results support the utility of pathway structures in determining pathway activities but also demonstrate the current limitations of the available structure-based tools. In the present comparisons, SPIA showed the best balance between consistency and the number of results and was relatively easy to use.Key PointsPathway methods using pathway topology found more significant pathways than methods not using pathway topology.Selection of the method has a large impact on the results.With clear cell renal cell carcinoma data, SPIA and CePa provided consistent results with different data sets.With type 1 diabetes data, all the methods made only few findings.Supplementary data are available online at http://bib.oxfordjournals.org/.


Multiple high-throughput molecular profiling by omics technologies can be collected for the same individuals. Combining these data, rather than exploiting them separately, can significantly increase the power of clinically relevant patients subclassifications.We propose a multi-view approach in which the information from different data layers (views) is integrated at the levels of the results of each single view clustering iterations. It works by factorizing the membership matrices in a late integration manner. We evaluated the effectiveness and the performance of our method on six multi-view cancer datasets. In all the cases, we found patient sub-classes with statistical significance, identifying novel sub-groups previously not emphasized in literature. Our method performed better as compared to other multi-view clustering algorithms and, unlike other existing methods, it is able to quantify the contribution of single views on the final results.http://neuronelab.unisa.it/a-multi-view-genomic-data-integration-methodology/.Our observations suggest that integration of prior information with genomic features in the subtyping analysis is an effective strategy in identifying disease subgroups. The methodology is implemented in R and the source code is available online at The online version of this article (doi:10.1186/s12859-015-0680-3) contains supplementary material, which is available to authorized users. Stratifying patients into distinct subgroups can lead to more accurate diagnostic and treatment strategies. Current methods for patient stratification are usually based on gene expression data and apply cluster algorithms to identify groups of patients having similar expression profiles –3. For eMethods for clustering multiple data layers can be grouped into three main categories, namely early, intermediate, and late integration. Early integration methods directly combine all features into a single dataset –8; interA number of data integration approaches for patients subgroups discovery were recently proposed, based on supervised classification, unsupervised clustering or biclustering. These methodologies are called multi-view learning . ExampleIn this study, we propose a new computational framework for multi-view clustering that aims to combine dimensional reduction, variable selection, clustering (for each available data type) and data integration methods to find patient subtypes, as described in Fig. http://cancergenome.nih.gov/). Our comparison experiments suggest that our method outperforms other existing integration methods, such as Tw-Kmeans  is the contribution of view i to the final multi-view cluster j.The proposed methodology for the analysis of multi-view biological datasets takes in input Prototype Extraction: for each view, the features were filtered by variance and clustered in order to find prototypes, reducing the input data dimension.Prototype ranking: the prototypes found in the step 1 were ranked based on their ability to separate the classes.Single view clustering: in each view, the samples were clustered using the prototypes created in the steps 1 and 2 as featuresIntegration: single view clustering results were integrated with a late integration approach, in order to obtain the k final multi-view meta-clustersThe approach consists of four main steps as shown in Fig. The late integration methodology can be considered as a further step of the proposed data mining pipeline, in which the clustering results of each single view are unified. This approach offers a number of significant advantages: (i) clustering algorithms can be optimally chosen with respect to each single view; (ii) it can be naturally parallelized; (iii) representation issues are avoided since clustering results are the inputs to the integration algorithms.The features with low variance across the samples were eliminated. Therefore the data were clustered with respect to the patients and the cluster centroids were selected as the prototype patterns. The centroid of each cluster was selected as the most correlated element with respect to the other elements in that cluster. Different clustering algorithms were used: Pvclust , SOM 2222, hieraThe idea is to evaluate several popular clustering techniques and compare their behaviour on the different views with respect to the hierarchical method that is the standard algorithm used to cluster genes. As noted in , clusterIC is the complete diameter measure, representing the average sample correlation of the less similar objects in the same cluster; EC is the complete linkage measure, representing the average sample correlation of the less similar objects for each pair of clusters; S is the singleton factor and CG is the compression gain. The evaluation function was defined in order to obtain the output value normalized between 0 and 1. The complete diameter and the complete linkage measures were calculated with the R “clv” package ). The samples in this dataset correspond to breast cancer patients with invasive tumors. Genomic data for two views were downloaded: RNASeq and miRNASeq (Level 3). Because level 3 data corresponds to already preprocessed data, only the batch effect was removed by the comBat method in the R “sva” package ) and publicly available data for two views were downloaded: gene expression and miRNA expression. Also clinical data was retrieved. The patients were divided info four classes: Classical, Mesechymal, Neural and Proneural as described in [Glioblastoma cancer dataset from the TCGA repository. The samples in this dataset correspond to glioblastoma patient with invasive tumors. TCGA website was accessed (ribed in .https://tcga-data.nci.nih.gov/tcga/ - Ovarian serous cystadenocarcinoma [OV]). The samples in this dataset correspond to patient affected by ovarian serous cystadenocarcinoma tumors. Publicly available data for three views were downloaded: gene expression, protein expression, and miRNA expression. Clinical data were downloaded in order to classify patients in three categories. In particular patients were classified by clinical stage: first class: stage IA, IB, IC, IIA, IIB and IIC, second class: IIIA, IIIB and IIIC, third class Stage IV.Ovarian cancer dataset from the TCGA repository (http://cbio.mskcc.org/). The samples in these datasets correspond to patient prostate cancer tumors. The MSKCC Cancer Genomics data portal (http://cbio.mskcc.org/cancergenomics/prostate/data/) was accessed and data for five views were downloaded: clinical data, gene expression, microRNA expression and copy number variation. Patients were classified in two classes by using clinical data by the tumor stage: class one is Tumor Stage I and class two is Tumor Stage II, III and IV. Classification of patient was done according to a previous study performed on the same dataset [Prostate cancer dataset from a study performed at the Memorial Sloan Kettering Cancer Center ( dataset .


Principal component analysis is used to summarize matrix data, such as found in transcriptome, proteome or metabolome and medical examinations, into fewer dimensions by fitting the matrix to orthogonal axes. Although this methodology is frequently used in multivariate analyses, it has disadvantages when applied to experimental data. First, the identified principal components have poor generality; since the size and directions of the components are dependent on the particular data set, the components are valid only within the data set. Second, the method is sensitive to experimental noise and bias between sample groups. It cannot reflect the experimental design that is planned to manage the noise and bias; rather, it estimates the same weight and independence to all the samples in the matrix. Third, the resulting components are often difficult to interpret. To address these issues, several options were introduced to the methodology. First, the principal axes were identified using training data sets and shared across experiments. These training data reflect the design of experiments, and their preparation allows noise to be reduced and group bias to be removed. Second, the center of the rotation was determined in accordance with the experimental design. Third, the resulting components were scaled to unify their size unit.The effects of these options were observed in microarray experiments, and showed an improvement in the separation of groups and robustness to noise. The range of scaled scores was unaffected by the number of items. Additionally, unknown samples were appropriately classified using pre-arranged axes. Furthermore, these axes well reflected the characteristics of groups in the experiments. As was observed, the scaling of the components and sharing of axes enabled comparisons of the components beyond experiments. The use of training data reduced the effects of noise and bias in the data, facilitating the physical interpretation of the principal axes.Together, these introduced options result in improved generality and objectivity of the analytical results. The methodology has thus become more like a set of multiple regression analyses that find independent models that specify each of the axes. X be a sample-by-item matrix, containing items with zero mean. The matrix can be factorized into one diagonal and two unitary matrices, X = UDV*. The unitary matrices U and V control the directions of the principal axes, while the diagonal matrix D records the singular values. The principal components for the samples in X are defined as Ys = XV (= UD). The unitary matrix V is frequently used as loadings for the components; however, the principal components for the items, Yi = X*U (=VD), will be used here instead.Principal component analysis is frequently used to transform matrix data consisting of multiple measured items. To represent the data efficiently, the first principal axis is fitted to the matrix, minimizing the distances between the data and the axis; then the next axis perpendicular to the first axis is fitted to the data, and so on -3. SinceX, even though these samples may have different weights and dependencies. Hence, noise tends to affect the resulting components. Skinner et al. (1986) identified this instability in the directions of the axes in the context of sampling design; since experimental designs often consist of repeated measurements, the effects of the weights and independencies among the data should become larger [As discussed below, the principal component methodology is unsuitable for the analysis of experimental or diagnostic data. One reason is that the method is sensitive to noise and bias in the samples; the method is intrinsically weaker with noise compared to factor analysis, which contains error terms. Additionally, because of how distance is defined , both sit al. 198 identifie larger . HoweverAnother disadvantage is with the method's generality; the identified principal components are only valid for the given set of data. A comparison or integration of principal components requires specialized assumptions, for example, that the experiments or observations to be integrated share common data, and that the experiments differ only in their means ,3. IndeeX was determined in accordance with the experimental design. The de facto standard of the center is the sample means, i.e., the item-wise means of all the samples; however, these means are not always the best choice because they are sensitive to the biases in the groups. Rather, many experimental designs have a control group that is suitable as the center. The introduced options for the training data and the center are used to improve the fidelity of the methodology to the design of experiments. With these revisions, the components present differences of the samples from the control group, but not the data fluctuations estimated by the standard deviation of data. Third, the effects of the size of X on the components were eliminated. Two categories of principal components, i.e., that of samples and items, are used to present the data, instead of the original biplot. Both categories are closely related, because one is the source of the other, and they can be represented on identical axes. Taken together, each axis would assign an independent multiple regression model. On an axis, differences from the control group are presented from a unique perspective.Here, modifications to the method are introduced which allow generality to be achieved in the results. First, the processes of identification and application of the axes were separated, enabling the axes to be shared among experiments. The unitary matrices that assign the axes are discovered using training data, which consist of representatives of the groups in the experimental design. A representative is found as the mean of a group, such as repeated measurements of a treatment. By taking means of repeats, the level of measurement error is reduced. By using robust methods such as the trimmed mean or median, the effect of outliers can be further reduced. In addition, by assembling sets of data into appropriate groups, bias due to unbalanced data among the groups can be removed. Second, the center of the data D, is not necessarily required; hence, the unitary matrices could be used instead of the principal components [D is used, because microarrays are often used for experiments or examinations in which a set of data contains groups of several repeated measurements, and these groups are frequently biased. As microarrays are often used in large projects, the resulting knowledge should be shared widely and integrated seamlessly. Here, the effects of the introduced options are observed in the separations of the groups and their robustness to noise. Additionally, the appropriateness of the cancellation of the size effect becomes obvious by the constancy in the range of the components presented. There also exists the possibility of sharing axes, which enables the use of a common framework among experiments.The effects of these modifications were observed in two microarray experiments. Since the microarray measures a very large number of items, e.g., the expression level of genes, methods to summarize the data are beneficial . Indeed,mponents . NeverthThis work used experiments that investigated several groups using repeated measurements. The data appeared in the Gene Expression Omnibus : a time The matrix of expression levels is subtracted using the reference data,s is the normalized sample data, r is the reference found for each item, and n and m are the number of samples and items, respectively. The reference, which determines the origin of the principal components, were found as the item-wise means of all the samples for the time-course data, or of the control group for the toxicology data. Missing data in X were replaced with zero; the same replacement was performed to remove certain items, such as negative genes in the analysis of variance test. This replacement is based on a fail-safe design: as zero elements in X indicates that there is no difference from the reference, the replacement of certain values with zero tends to move the resulting principal components closer to the origin, but not distally, which may suggest positive results.where X, the number of functional items, mf, could differ among samples due to missing data; hence, differences in mf should not affect the size of the principal components so that comparability is maintained. Such differences in mf would be much larger when comparing results obtained from different sets of items, for example, whether test-negative genes are included or not. However, the size or magnitude of the principal components depends on mf, as follows.In y be an element of the principal components for samples, Ys, of a matrix X that contains mf functional items, and v be an element of the unitary matrix V derived from X. For the j th component of the k th sample, kj y∝ mf1/2.Theorem 1. Let V is a unitary matrix, V*V = I, thus j. Hence, j v∝mf -1/2. According to the definition of principal components, the value of k yfor the j th component is calculated as kx. We can assume stability of Proof. Since mf, the components should be scaled. After a singular value decomposition is performed on the matrix with mf functional items, the scaled components for the samples are estimated asTo maintain the principal components' compatibility regardless of the size of D, the ratios among the axes will be retained. In the same manner, the ratios for the items are scaled as Zi = nt-1/2Yi, where nt is the number of data samples of the matrix in which U was found.according to Theorem 1, this will unify the magnitude of PC. Although scaling in principal component analysis tends to be employed in the context of the pre-scaling of data or multidimensional scaling for biplots ,3, the mT was the set of the sample means from each group, for example, each point in the time-course data or each chemical in the toxicology data, subtracted with the reference as in (1). The axes were found using singular value decomposition, T = UTDTVT*. The principal components for the samples are defined as Ys = XVT, and those for the items are defined as Yi = T*UT. Additionally, the effects of robust alternatives to singular value decomposition algorithms were observed using two different functions in R : [robustSvd function  [pcaMethods library [PcaHubert function  in the rrcov library [Ys or Yi. The obtained components were scaled as in the previous section. The source code for R is available as Additional file The axes were identified in a set of predetermined training data matrices that were intentionally chosen. Unless otherwise stated, the training data , 2011):  the robu., 2003)  in the p library  and the  library . In the X , with a view towards group separation . In the X Figure , the sepX Figure  or for tX Figure . The samX, the group separation degrades  measured three toxic chemicals , three nontoxic chemicals , and one nontoxic control; there were five repeats . BesidesVT, for estimating the principal components for unknown samples. The training data was prepared using all sample means except those from group 3. The obtained first axis successfully separated the group 3 samples from the nontoxic chemicals, and the second axis separated the toxic chemicals DV*, where 0<c <1; they are complementary and can reproduce information for X [Ys and Yi could differ largely especially when the numbers of samples and items are distinctively different; however, those in the scaled Zs and Zi will become similar, hence they can be presented together.Another benefit to the scaling is that it allows biplot-like presentations. According to Jackson's definition, biplots present on for X . HoweverV, multiply the gene items of a sample stored in a row of X, to find the target estimation of the regression function. The estimation of the component of a sample is stored in the corresponding column of Ys and Zs. The contributive gene item gets a high score in the corresponding columns of V, Yi, and Zi. However, this methodology is not purposive in finding out certain relationships of interest, unlike in multiple regression analysis. It represents relationships between groups rather objectively. Therefore, the analyst has to ascertain the proper axis by observing separation of the group scores.By separating finding and applying the axes, the methodology has become analogous to a set of multiple regression analyses; each of the axes shows an independent regression model. On an axis, principal components for group representatives show which target is estimated. The principal component for a sample is a result of the regression; it shows the tendency of the sample to the estimated target. Components for the items show which items are contributive for the regression model. For example, PC1 for the mammary gland development data, which is assigned on the X-axis in Figure The authors declare that they have no competing interests.R scripts. The complete scripts to import, analyze, and output the data.Click here for fileResults of principal components for samples, calculated by using other methods or conditions. E. Results of the robustSvd function [F. Axes were found in 5,892 positive genes with the analysis of variance test. G. Results of the PcaHubert function [H. Axes were found in the 5,892 positive genes. I. Effect of pre-scaling on the data. The whole set of 12,487 genes of centered data were divided by standard deviation before subjected to PCA. J. Axes were found in the 5,892 positive genes.function  to the wfunction  to the wClick here for file


MicroRNAs are now increasingly recognized as biomarkers of disease progression. Several quantitative real-time PCR (qPCR) platforms have been developed to determine the relative levels of microRNAs in biological fluids. We systematically compared the detection of cellular and circulating microRNA using a standard 96-well platform, a high-content microfluidics platform and two ultra-high content platforms. We used extensive analytical tools to compute inter- and intra-run variability and concordance measured using fidelity scoring, coefficient of variation and cluster analysis. We carried out unprejudiced next generation sequencing to identify a microRNA signature for Diabetic Retinopathy (DR) and systematically assessed the validation of this signature on clinical samples using each of the above four qPCR platforms. The results indicate that sensitivity to measure low copy number microRNAs is inversely related to qPCR reaction volume and that the choice of platform for microRNA biomarker validation should be made based on the abundance of miRNAs of interest. MicroRNAs (miRNAs/miRs) are 20–22 nucleotide long RNA molecules that are important negative regulators of protein-coding gene expression. They are transcribed by RNAPol-II245891011131415171819Currently, quantitative PCR (qPCR) is the favored method for determining miRNA expression, due to its accuracy, simplicity, reproducibility and lower cost than other hybridization or sequencing-based technologiesHere, we compare four different qPCR platforms from two different vendors, with specific focus on circulating human (plasma) microRNAs. We test the performance of these platforms in validating a microRNA signature for diabetic retinopathy (DR), identified in clinical samples using a discovery approach. We examine both cell and serum RNA isolations on the ViiA7 (Life Technologies), TaqMan Low Density Array , OpenArray (Life Technologies) and Dynamic Array (Fluidigm) systems. The ViiA7 employs the standard 96-well plates (5 μl reaction volume described earlier)25An overview of the different platforms  comparedThe raw cycle threshold (CT) values generated on each platform are plotted in To assess the variability of each platform, the co-efficient of variation (CV) was calculated for each microRNA assay . The medAs an indicator of overall reproducibility, the fidelity of each platform was assessed. Fidelity was calculated as the percentage of replicates that differed by less than 1, 2 or 3 CT values and is presented as deviation from fidelity in CT values were then segregated into expression levels and the replicate variation plotted . Ultra-hTo determine how reproducible the high-throughput technologies are, RNA extracted from human islet cells and serum was run twice on each platform by two different users (RJF and MVJ). The average difference between the two sets of replicates is plotted in One of the ultimate goals of most miRNA expression studies is to determine a biomarker signature related to particular pathology. To assess a biomarker signature for diabetic retinopathy (DR), we used unbiased small RNA sequencing of plasma samples from diabetic individuals with or without retinopathy. While analyzing samples from subjects with (DR) and without  diabetic retinopathy  we selec2 values 0.88 and 0.66 for OA and DA respectively. The TLDA platform had an R2 value of 0.91  in four human subject samples  and plotted these as pair-wise comparisons against ViiA7 (“gold standard”) platform for OA, DA  and TLDA of 0.91 . The lowCluster analysis was implemented to determine the ability of these qPCR platforms in allocating miRNAs into distinct clusters. Each platform grouped the microRNAs into four distinct clusters . Again, The appeal of high-throughput technologies is readily apparent; they provide a significant decrease in sample processing time and reagents consumed, with a substantial increase in the volume of data generated in the same time. For researchers within the burgeoning field of miRNA biomarker research, such a tool is immensely desirable, especially when dealing with a multitude of clinical samples from large trials. Such assay performance would also be required should a miRNA profile related to diabetes, reach clinical practice. It is imperative that technologies used for quantitative research or in clinical practice be accurate, sensitive and reproducible, and high-throughput platforms must present a similar (if not greater) level of reproducibility offered by current high sensitivity/low-throughput platforms.The two high-throughput technologies discussed here represent the two most contemporary high-throughput qPCR platforms. The Dynamic Array platform offers a finely crafted plate with a set of individually pressure-tested valves that control the flow of reagents into the microfluidic plate. The OpenArray platform provides a high density PCR plate/slide that allows automated sample loading using a specially designed robotic device (Accufill™). The major difference is that the OpenArray comes with pre-printed miRNA assays while the Dynamic Array provides flexibility to choose the miRNA assays just before setting up the plate. The DA platform thus provides only the “hardware” for users to perform qPCR. Although this flexibility is an advantage, the limitation of the DA is that it heavily depends on the compatibility of the pooled RT primers with the TaqMan real-time PCR assays. Of the 48 miRNAs selected from the TaqMan primer pool A  six of these assays could not be read on the DA due to incompatibility between the RT and PCR primers. Since the proprietary primer pools are compatible with the primers printed in TaqMan Low Density Arrays (TLDAs) or OpenArray plates only, the individual assays available from Life Technologies may not be compatible with the multiplex RT pools. Of course, users can avoid this by ordering customized primer pools from Life Technologies to assess a specific set of miRNAs.In our analysis, the OpenArray system was the most reproducible high-throughput platform tested, with less inter- and intra-run variation than the Dynamic Array. This conclusion correlates with a recent study comparing these platforms for screening genomic mutations in the Ashkenazi Jewish communityOur data demonstrates that replicate variability was exacerbated when quantifying low abundance transcripts on any of the platforms. As shown in For medium-throughput studies, researchers can utilize the TLDA platform. We found a comparable level of reproducibility on this platform when using cards from the same lot , but havCorrelation between technologies is required to translate relative gene expression data on different qPCR platforms. Since CT values cannot be directly compared across platforms due to the differences in sample preparation, reaction volume and detection technologies, correlation between z-scored transformed data  is a relSegregation of miRNAs based upon their expression profile is useful for biomarker identification. Our cluster analysis  demonstrThe miRNA signature profile generated by our smallRNA sequencing, although representative, demonstrates the importance of rigorously analyzing new technologies, especially those that may become routine for biomarker discovery and validation and potentially be used in clinical practice. This report demonstrates the importance of multiple technical factors that can influence the conclusions derived from miRNA qPCR data. Although there are differences in the respective detection systems , the size of the reaction may also affect the reproducibility of these high-throughput systems. Although both platforms have a reaction volume in nanoliters, the OpenArray reaction volume is 2.2-fold more than the Dynamic Arrays. During the optimization of our standard (96-well) qPCR analysis for detection of low-copy miRNA/mRNA transcripts, the amplification curves differed by 0.5 cycles when the reaction volume was reduced from 10 μl to 5 μl(data not shown). Considering the Poisson distribution of miRNA templates, this difference can be significant when more than two-fold difference is factored in, even at the nanoliter range. Although each of the platforms discussed have their own advantages/disadvantages, users need to consider all aspects before choosing an analytical platform, and ideally should test their data across another high-sensitivity platform such as the 96-well platform. These observations provide a useful guide for researchers planning to switch to high-throughput qPCR technologies for quantitative assessment of gene transcripts in large number of research/clinical trial samples. Furthermore, the development of a sensitive, reliable and minimally invasive test with high predictive power for clinical stratification of DR will provide a surrogate marker for clinical trials, improve prognosis information and guide development of future therapies aimed towards preventing the vascular complications of diabetes.The study was approved by the University of Sydney Ethics Committee and all samples were collected following written informed consent. Total RNA was isolated from human islets (Islet Transplant Consortium) and from serum samples, collected from individuals with or without diabetes. The study was carried out in accordance with the approved guidelines. Samples were tested for a total of 48 miRNAs, selected based upon their expression in small RNA sequencing, their relevance to islet biology or diabetes, or their suitability as housekeeping transcripts. However, six of these 48 miRNAs were identified to be incompatible on the Fluidigm system (see discussion) and therefore all of the analysis for the two platforms is based on comparison of the expression of these 42 miRNAs on all the platforms tested. The 42 miRNAs include three “housekeeping” miRNAs . Each PCR reaction was undertaken at least in duplicate. Technical replicates were also included within the same run as well as in different runs, and repeated by two or three users . No significant variation was seen in data obtained by the three users.To aid comparison with the pre-amplified samples run on the high-throughput technologies, samples were pre-amplified and run in the 96-well format using 5 μl reaction volumes. Reverse transcription (RT) and pre-amplification (PA) was completed using the Megaplex RT/PA Primer Pools (Life Technologies) using the manufacturer’s protocol, with minor changes. Briefly, each sample had an RNA input of 100 ng (measured by Nanodrop), went through 12 cycles of PA and was diluted 1:40 in nuclease-free water. qPCR was undertaken in 0.1 ml optically clear 96-well plates using TaqMan MicroRNA Primer/Probe mixes and TaqMan Fast Universal PCR Master Mix (2X), No AmpErase UNG (Life Technologies).RT and PA were undertaken with Megaplex RT/PA Primer Pools using the manufacturer’s OpenArray microRNA Panel protocol as described elsewhereRT and PA were undertaken with Megaplex RT/PA Primer Pools using the protocol supplied by Fluidigm Corporation. Each sample (input 100 ng), underwent 16 PA cycles (recommended by the manufacturer), then was diluted 1:10 in nuclease-free water, combined with TaqMan MicroRNA Primer/Probe mixes and TaqMan Fast Universal PCR Master Mix (2X), No AmpErase UNG, and loaded onto a 48 × 48 Dynamic Array IFC (Integrated Fluidic Circuit) using the IFC Controller MX. qPCR was performed on the BioMark system (Fluidigm Corporation).™ chip and sequenced on Ion PGM using the Ion PGM™ Sequencing 200 Kit v2 . The aligned BAM files were analyzed using Strand NGS (Strand Life Sciences) and differential miRNA expression was determined by a two-fold change in normalized read number.RNA was isolated from plasma samples using protocol described earlierThe data were analyzed using Statistica for Windows analytical software . For intra-platform reproducibility analysis the coefficient of variation (CV%) was calculated between replicate measurements. Data was obtained from the multiple measurements  of 40 transcripts in four samples. miR-140 and miR-320 were excluded due to the lack of results on TLDA and DA platforms respectively.CT/CRT values could not be compared directly so they were transformed into Z-scores within each platform. Pearson correlation and linear fits were calculated. Level of compression or expansion of data measured on the different platforms were evaluated by comparing the slope of the best fitted line of a least square linear regression of the Z-scores between pairs to the “ideal” slope of 1.Ward’s linkage method and Euclidean distances were used for cluster analysis. Separate dendrograms were generated for each analytical platform. Number of clusters was based on Mojena rule or distances between the dendrogram knots. In each analytical method four clusters were observed. The miRNAs of each cluster were then calculated using k-means method. MicroRNAs with >15% missing data  were excluded from the calculations; otherwise missing data were replaced by mean. Clusters concordance was estimated by comparing the clusters generated by other platforms with the ViiA7 (“gold standard”). The assignment was declared in concordance if at least two miRNAs were placed in the same cluster by different methods.How to cite this article: Farr, R. J. et al. A comparative analysis of high-throughput platforms for validation of a circulating microRNA signature in diabetic retinopathy. Sci. Rep.5, 10375; doi: 10.1038/srep10375 (2015).


Sequencing-based gene expression methods like RNA-sequencing (RNA-seq) have become increasingly common, but it is often claimed that results obtained in different studies are not comparable owing to the influence of laboratory batch effects, differences in RNA extraction and sequencing library preparation methods and bioinformatics processing pipelines. It would be unfortunate if different experiments were in fact incomparable, as there is great promise in data fusion and meta-analysis applied to sequencing data sets. We therefore compared reported gene expression measurements for ostensibly similar samples  in several different RNA-seq studies to assess their overall consistency and to examine the factors contributing most to systematic differences. The same comparisons were also performed after preprocessing all data in a consistent way, eliminating potential bias from bioinformatics pipelines. We conclude that published human tissue RNA-seq expression measurements appear relatively consistent in the sense that samples cluster by tissue rather than laboratory of origin given simple preprocessing transformations. The article is supplemented by a detailed walkthrough with embedded R code and figures. Standard RNA-seq experiments begin with RNA extraction, followed by library preparation, amplification and sequencing . SubsequIn recent years, the availability of RNA-seq data sets has dramatically increased, and today several resources are available on the Web where users can download either raw sequence data or precomputed gene expression values, or in some cases query specific genes of interest in tissue-specific contexts. The continuously increasing amount of public expression data promises new opportunities to combine information from different studies, whether in the form of meta-analysis studies (such as genome-wide association studies)  or by trHowever, publicly available RNA-seq data sets from different laboratories and studies are sometimes claimed to be incomparable owing to the combined influence of reagent batch effects, differences in RNA extraction protocols, library preparation methods and computational processing  from different sources are consistent, whether any particular preprocessing transformations are necessary and whether one needs to reprocess all samples in a consistent way from raw sequence FASTQ files instead of working from published FPKM/RPKM values. We also attempt to quantify which factors contribute the most to variability between samples, using principal component analysis (PCA) and analysis of variance (ANOVA). The main condition we use to discuss ‘consistency’ is to see whether the tissue samples  from several different studies, when subjected to PCA, cluster primarily by tissue rather than by study, or according to some other principle.Our results indicate that RNA-seq expression profiles in the human tissues studied here appear relatively consistent in the sense that they can readily be made to cluster according to tissue with only light preprocessing.The best results are obtained by log transformation and application of a batch effect correction method . There et al., which we call AltIso as an abbreviation of the title of the corresponding paper  and brain genes  are also found among the most contributing genes.Supplementary Figure S1a).If, instead of looking at gene loadings, we look at correlations between principal component scores and study-dependent experimental factors, we find that the largest variance component  mainly reflects tissue and the second one  mainly correlates with library preparation method and study  are different when taken ‘as is’, but one might expect the differences to level after selecting only those 13 323 genes for which an expression value was reported in all four studies. However, the distributions of the expression levels of these common genes  still look different between the studies, similar to the original raw data. This picture is consistent across tissues.A potential reason for the difficulties in clustering the raw data from different studies is that the F/RPKM distributions could have different distributions. As shown in Supplementary Figure S2a–f, S3).Different RNA extraction methods are naturally expected to result in differences in the distribution of RNA molecules. Among the studies in this comparison, RNA Atlas is the only one that used ribosomal RNA depletion , which leads to a different proportion of mRNA molecules versus noncoding RNA molecules in the sample (the rRNA depleted samples are expected to contain considerably more noncoding RNAs). It appears that the original RNA-Seq Atlas paper calculated RPKM values by dividing counts by the total number of mapped reads (including both non-protein coding and coding genes), which would lead to protein coding genes getting systematically lower RPKM estimates than a poly-A-enriched sample owing to the presence of noncoding RNAs ‘competing’ for the sequencing reads. The published data for RNA-Seq Atlas samples consistently show lower RPKM values than other samples The fact that distributions greatly differ suggests that a quantile normalization, which attempts to adjust distributions to become more similar, might be able to recover a good clustering by tissue. However, this was not the case for the raw published data , ‘Read length’ , ‘Preparation’ (RNA extraction method), ‘Total number of raw reads’, ‘Study’ and ‘Tissue’ were included . None ofSupplementary Figure S1d). To further confirm that PCs 2 and 3 describe general tissue specificity, we performed a type of cross-validation by first applying PCA to log2-transformed values from all data sets except one (AltIso) and then projecting the AltIso expression profiles onto the obtained components PC2 and PC3. As shown in Supplementary Figure S2A–F, Supplementary Figure S3.) When the F/RPKM values are log transformed, the magnitude differences will be decreased and the importance of the preparation factor for modeling each gene expression estimate will appear smaller relative to other factors.Logarithmic transformation is known to transform RNA-seq data into a more normally distributed shape with less dependence between the mean and the variance, as well as dampening the effects of extreme outliers . When coSupplementary Figure S4C–D.)We also attempted a quantile normalization followed by PCA on the log-transformed values, but were unable to improve on the clustering seen before normalization .As mentioned before, the three samples derived from the RNA-seq Atlas project are often distinct from the rest of the samples and tend to cluster together rather than with the corresponding tissue type, presumably because the RNA in these samples was purified using rRNA depletion instead of with poly(A) selection of mRNA that was used for all the other samples. The separation of the RNA-seq Atlas samples from the rest exemplifies how different procedures in sample preparation and following sequencing steps can affect the data. In the PCA analysis of log-transformed FPKM values, although the samples cluster better according to tissue type than the unprocessed values, they still appear to be ordered in a consistent pattern according to which study they derive from, indicating again that there are systematic technical effects from all four studies that need to be considered. For instance, the RNA-Seq Atlas samples from each tissue are further away from the tissue-cluster centers than the other samples c; they aTo adjust for biases introduced in these data sets owing to differences in the library preparation and sequencing for the different studies, we used the ComBat function included in the sva R package for removal of batch effects, only using information on what study the samples derive from .As shown in We conclude from the above analyses that published precomputed expression levels for human tissue RNA-seq data from different sources are poorly comparable at a global level and that log transformation and modeling of known batch effects are essential to make the data comparable as consistent reference data. Next, we examined whether the results could be improved by reprocessing the data from raw FASTQ sequence files.Supplementary Figure S1b).To remove potential bias from the fact that different bioinformatics pipelines were used to calculate published FPKM/RPKM values, we repeated the same analyses as above using FPKM values obtained from reprocessing raw data (FASTQ files) in a consistent way. We used human tissue RNA-seq data from five studies that had published FASTQ files: BodyMap, Evolution of Gene Expression, Human Protein Atlas, RNA-seq Atlas and AltIso , 14, 15.The genes with highest loadings in the first three principal components were various mitochondrially encoded genes with relatively high FPKM values (up to hundreds of thousands in FPKM) in all samples, together with a few tissue-specific genes, for example, APOE (ENSG00000130203), ALDOB (ENSG00000136872) and YBX3 (ENSG00000060138) that are highly upregulated in kidney . For details, see accompanying walkthrough. To avoid this type of noise from highly expressed mitochondrially encoded genes, a masking option for mitochondrial genes can be used when running Cufflinks. ANOVA analysis was again performed, and as shown in Supplementary Figure S1e.) Logarithmic transformation again serves to improve the clustering of tissue types to some extent, but there is still systematic variation in the raw data that remains problematic even after removal of potential bias from differences between bioinformatics pipelines. To overcome the problem with laboratory batch effects in the raw data, we used the ComBat function for removal of known batch effects. As shown in The impact of log transformation was also investigated for the reprocessed FPKM values, resulting in a similar result as the published values. As shown in per se improve the overall consistency of RNA-seq expression profiles, although it does avoid the loss of genes associated with combining gene identifiers from different annotation systems.In summary, we conclude that the reprocessing of raw data does not A different way to address the question how much the choice of the bioinformatics pipeline  influences the results is to perform ANOVA and PCA correlation analyses as described above but adding the quantification as an additional factor. We cannot do such an analysis for the published data alone, because each study used a different bioinformatics pipeline, and thus the effects of the pipeline would be impossible to distinguish from other factors that varied between the studies. We therefore combined both the published and reprocessed data and performed ANOVA with the new factor ‘quantification’ that represents the mapping and quantification steps. All reprocessed samples were quantified with the Tophat/Cufflinks pipeline, as were the HPA samples in the published data. The ANOVA on untransformed F/RPKM values k indicatSupplementary Figure S1c, f) indicates that the main directions of variation in the data are not strongly correlated to the quantification method.This result is in apparent conflict with the finding above, namely, that reprocessing data from FASTQ in a consistent way does not improve the clustering of the samples by tissue (which it would be expected to do if different quantification methods introduced systematic bias.) We speculate that the differences in quantification methods contribute to gene-to-gene variation and unspecific noise, but not to systematic variation that would affect samples from separate tissues in different ways. An analysis of the correlations of the first two principal components to the experimental factors . With the findings of this study, we conclude that publicly reported precomputed values for gene expression (FPKM/RPKM) are not comparable at a global level in their untransformed state. However, after log transformation and removal of batch effects, the data show global consistency. Logarithmic transformation alleviates problems in clustering of the three tissue types, but is still not sufficient to enable the most of the variance to be explained by tissue type. The largest variance is explained by known or unknown study-specific effects that will disturb clustering unless they are identified using statistical modeling, and removed before analysis. Today, numerous methods are available for bias detection and removal and in this study, ComBat was successfully used for that purpose, although one of several alternative methods could have been used . ReproceThere are many potential alternatives to the various RNA-seq data processing steps, but we have confined this study to a relatively standard workflow for the sake of clarity. Potential improvements include alternative methods for variance stabilization transformations (in contrast to the log transformation used in this case) and other normalization methods, for example, GC content correction methods and scaling methods like Trimmed mean of M values. The choice of PCA as the visualization method was made based on its popularity, but it is possible that other methods such as multidimensional scaling, nonnegative matrix factorization or t-distributed stochastic neighborhood embedding could have yielded better results. Additionally, it would be interesting to use a different set of tissues that are more similar in their expression profiles compared with the three tissue types used in this study. Nevertheless, we expect that the findings of this study will play an important role in the continuous efforts to secure the usefulness of public RNA-seq data and fulfill the requirement of reliability in comparisons between data sets generated in different laboratories.Supplementary data are available online at http://bib.oxfordjournals.org/. In addition, a walkthrough of the analyses in the article (as well as many variations not reported in the article in the interest of clarity) with embedded code and figures is available at https://github.com/hussius/publicRNA seqdata.Publicly available data sets with precomputed RNA expression levels are not comparable in their untransformed state in the sense that samples from the same tissues obtained in different experiments do not cluster by tissue.Logarithmic transformation improves clustering of samples in principal components 2 and 3, while principal component 1 still seems to be dominated by study-specific factors.RNA extraction method, read length and sequencing layout (single-end versus paired-end) contribute strongly to variation between samples.Removal of known batch effects is essential for clustering based on tissue type.Reprocessing raw data avoids loss of expression information because of gene identifier matching issues but does not serve to improve clustering.to M.H.); EU FP7 STATegra 306000 (to D.G.C.); EU FP7 Affinomics 241481 (to F.D.).Knut and Alice Wallenberg Foundation (