Molecular measurements from cancer patients such as gene expression and DNA methylation can be influenced by several external factors. This makes it harder to reproduce the exact values of measurements coming from different laboratories. Furthermore, some cancer types are very heterogeneous, meaning that there might be different underlying causes for the same type of cancer among different individuals. If a model does not take potential biases in the data into account, this can lead to problems when trying to predict the stage of a certain cancer type. This is especially true when these biases differ between the training and test set.We introduce a method that can estimate this bias on a per-feature level and incorporate calculated feature confidences into a weighted combination of classifiers with disjoint feature sets. In this way, the method provides a prediction that is adjusted for the potential biases on a per-patient basis, providing a personalized prediction for each test patient. The new method achieves state-of-the-art performance on many different cancer data sets with measured DNA methylation or gene expression. Moreover, we show how to visualize the learned classifiers to display interesting associations with the target label. Applied to a leukemia data set, our method finds several ribosomal proteins associated with the risk group, which might be interesting targets for follow-up studies. This discovery supports the hypothesis that the ribosomes are a new frontier in genadaptivelearninge regulation.https://github.com/adrinjalali/Network-Classifier/tree/v1.0.We introduce a new method for robust prediction of phenotypes from molecular measurements such as DNA methylation or gene expression. Furthermore, the visualization capabilities enable exploratory analysis on the learnt dependencies and pave the way for a personalized prediction of phenotypes. The software is available under GPL2+ from The online version of this article (doi:10.1186/s12864-016-2647-9) contains supplementary material, which is available to authorized users. Over the past few decades, biology has transformed into a high throughput research field, both in terms of the number of different measurement techniques as well as the amount of variables measured by each technique , and is more and more targeted to individual cells . This haL1-SVMs [In this work, we introduce a method that is aware of this potential bias and utilizes an estimate of the differences during the generation of the final prediction method. For this, we introduce a set of sparse classifiers based on L1-SVMs , where eL1-SVMs . HoweverL1-SVMs . One issL1-SVMs .We show that this method exhibits state-of-the-art performance for different cancer types, with gene expression or methylation data sets as the input. Since the weighting of the classifiers is customized for each test sample, the estimated confidences can offer insights into the specific characteristics of each individual’s cancer. To facilitate interpretation of the model, we then create a visualization of the important genes found through this analysis for each test sample. Additionally, we show how the important genes of the training set can be found using our learning method and cross validation.Our idea might resemble ensemble feature selection, which involves aggregating multiple feature scores from several scoring mechanisms. These scoring mechanisms vary from being several different methods, to being the same method applied to different parts of the data such as a random cross validation scheme . This idg1 and g2 are both strong but redundant, individual learner 1 might include g1, and individual learner 2 might include g2. Also, if there are more redundant or related genes in the gene pool, they will be used to estimate how reliable g1 and g2 are. Therefore instead of dismissing them, we exploit the fact that they exist.Similar to this approach, in another work, iRDA uses a different approach and can report multiple parsimonious gene sets . One sigRelated to sorting genes and testing for significance of a reported gene set, Gene Set Enrichment Analysis (GSEA) and its modifications are a commonly used tool , 13. GSEAcute Myeloid Leukemia (LAML) [a (LAML) : At the Breast invasive carcinoma (BRCA) [a (BRCA) : This daa (BRCA) , and aloa (BRCA) , 21. ForIn this article, our method is applied to two different data types: gene expression data and DNA methylation data, which we retrieved from The Cancer Genome Atlas (TCGA) . TCGA isTo prepare gene expression data for analysis, microarray probes are mapped to their respective gene. If there are multiple probes for a gene, the median reported gene expression value of those probes is adopted as the gene expression for that gene.Preparing the methylation data, we use the nearby gene for each methylation site available for each sample and each methylation site. The median beta value of methylation sites mapped to each gene is taken as the methylation value of the corresponding gene. In this process only methylation sites located on the promoter region of a gene are considered and others are discarded.For a given prediction problem the idea of boosting is to find an optimal combination of classifiers, also called “weak learners” . There aWhen dealing with cancer, we need to consider the fact that tumors of the same type of cancer can be very different in nature and they are usually classified as different cancer subtypes. In fact, even one single tumor can be very heterogeneous . This meOur method can be characterized by the following key parts:Fit several individual classifiers to the data, in such a way that the features of the data they use are disjoint sets.Training phase: Estimating the reliability of input features of the classifier;Estimating the confidence of the output based on the decision values.Calculate the prediction confidence of each individual classifier by: Calculate a weighted prediction label based on the individual classifier confidences.Prediction phase: L1 regularized SVM for this purpose with a linear kernel [L1 regularization makes the SVM sparse, i.e. using only a few input features, and the linear kernel allows us to infer which features are used in the decision function of the SVM after it is fit to the data.A wide variety of classifiers is possible within our framework. One requirement is that the classifier is regularized . The classifier is also required to report the probability of its calculated output, or to give a decision value according to which it chooses the predicted class. We use an r kernel . The L1 The model starts with no individual classifiers and an empty set of excluded features. In each step, the excluded set of features is removed from the data, then a classifier is fit to the data. Next the features used by the most recent trained classifier are added to the excluded set. In the case of a linear kernel SVM, this is achieved by finding features with a non-zero coefficient in the model. This way the features being used by classifiers are disjoint and might represent different underlying causes of groups into which samples are to be classified.f, we try to choose a few statistically dependent features, and fit a model to predict f. To find these features, first the estimated maximal information coefficient (MIC) of all other features with feature f is calculated [% or the 5 features with highest MIC with f (if the top 5 % features consist of less than 5 features), are selected as predictors of f. Given a test sample, the closer the predicted value of f is to the observed value, the more reliable it is. To quantify this, we need to not only know the predicted value of the feature, but also a confidence interval for that prediction. This can be achieved using Gaussian processes, which give the mean and variance of the posterior probability under the condition of observed values for selected features. A weighted average of these values gives us the overall reliability of the features of an individual classifier.Given a set of classifiers, the question is how to combine them to come up with a joint prediction value for each test sample for which we want to predict the output label. The intuition behind combining the classifiers is to put more weight on classifiers that use features whose behavior is similar to the training data. This is motivated by the fact that some parts of the test data might behave very differently to the training data, meaning that a classifier using these features should have lower performance than a classifier using features that are distributed similarly to the training data. Therefore we need to evaluate the reliability of the input features of each individual classifier. In scenarios like gene expression or methylation analysis, we usually have many input features. Furthermore, many features are correlated and statistically dependent. The idea of our new method is to build separate prediction models for each feature of each classifier. These prediction models can then be used to obtain a confidence for the feature in a given test sample. These confidences can then be combined for each classifier to give a weighting of the classifiers for the given test sample. To evaluate an observed feature lculated . Then, fIn addition to the confidence in the classifier estimated by looking at the confidences of its individual features, we also account for the confidence that the classifier has in the prediction label of the test sample. If the method supplies such a confidence value , we can directly use it. Otherwise, we estimate it using the decision value. In our setting, the linear SVM gives a decision value whose sign defines the predicted class. Using these values we estimate a confidence for each individual classifier. Several approaches exist for deriving a confidence from the decision values . WhetherX to be the set of input samples, Xs to be the input vector of sample s, ys and s, Δ to be the set of individual classifiers, li to be an individual classifier, li, li(Xs) to be the label predicted by classifier li for sample Xs, and f to be a feature, Xs,f to be the observed value of feature f in sample Xs, f in the decision function of classifier li, and gf to be the Gaussian process predicting feature f using feature set Φf. Also gf under the condition of observing values of features in Φf, and li. Here F is the cumulative distribution function of a standard normal distribution.More formally speaking, define N is the number of individual learners to be included in the model, Φl is the union over all X after discarding all features of the set Φl. TOP is the function which selects the maximum of the top 5 and top 5 % features f′ of all features ordered by MIC with feature f.The training phase of the model is shown in Fig. Xs, the estimated confidence of a feature f is: Now given a test sample li is estimated as: Then the overall feature reliability or confidence of a classifier li is: Also the estimated output confidence of the classifier li is then: and the final confidence of the classifier Finally, the predicted class The interpretation of the model can be understood on two different ways. First we assume for a given training data set, the model is trained and a new test sample is given. For the given test sample it is possible to visualize the reliability of each used feature in individual classifiers, as well as the overall confidence of each individual classifier. Used features can be superimposed onto a PPI network as well as their reliability and the confidence of their respective individual classifier.s,t) is defined as the number of times the respective features have occurred together in an individual classifier. Then, all edges with low weights are discarded. In order to find a threshold to prune edges according to their weights, a Gaussian kernel density estimate is fit to the weights of the edges, and the threshold is chosen at the 90th percentile. Nodes that have an appearance frequency higher than the threshold are labeled by their gene names and edges having a higher weight than the threshold are kept in the graph.Gene expression and methylation level measurements from cancer samples are usually very noisy. Furthermore, cancers are usually very heterogeneous. Additionally, there might be different subgroups for each interesting group , for which the importance of the features also differs. To get a global picture of the important features, we therefore evaluate how often certain features are selected by the classifiers using 100 random train test partitionings with 80 % of the data for training and 20 % of the data for testing. To visualize high confidence relationships between features, we create a graph which has a node for every chosen feature in any of the 100 train partitions in any of the individual classifiers. The weight of an edge  package are taken. In the case of stochastic gradient boosting, the representing class is GradientBoostingClassifier, the number of classifiers is set to 100, and to make it sparse and prevent over-fitting, the maximum number of features for splits in trees is set to 5, and the maximum number of layers is set to 2. For AdaBoost, AdaBoostClassifier is used, which is an implementation of AdaBoost-SAMME [DecisionTreeClassifier with maximum depth set to 2, and the number of weak classifiers set to 100. Parameters of the two boosting algorithms are chosen by a grid search on their parameter space over all the data sets and selecting the parameter sets which give a robust and stable result over all experiments.To compare the performance of our method with other methods, the implementations present in Python st-SAMME , with weν-SVM with ν=0.25 is used, once with a linear kernel, and once with an RBF kernel; γ parameter of the RBF kernel is set to (num of features)−1. The ν parameter is set to the maximum value for which the optimization function is solvable with libsvm for all analyzed data sets [minepy package [As an SVM, ata sets . Smaller package .The PPI network used in our analysis is from the Human Protein Reference Database (HPRD) . Almost % of the data for training and 20 % of the data for testing. To compare the performance of the different methods, Area Under the receiver operating characteristic Curve (AUC) [The performance of the method was compared with that of two ensemble methods, AdaBoost and stochastic gradient boosting, as well as an SVM with linear kernel, and an SVM with an RBF kernel. We also included our implementation of the NICK method . We randve (AUC)  was calcve (AUC) , meaningHere we present the results of running the method on the TCGA-LAML gene expression data set.L_” represent individual classifiers, and other nodes are labeled with their respective gene name. The color of the node shows its confidence compared to other nodes; the darker the node, the higher the confidence. In the case of a gene, it is the confidence or reliability of the feature (cf), and in the case of an individual classifier, it is the overall estimated confidence  is (are) influencing the prediction most. To this end, individual learners as well as the features they use are visualized as in Fig. 3To get a better overview of the individual features that were chosen by the classifiers for the particular test sample, we visualized the corresponding genes on a graph containing information about the PPI network in Fig. 3Most of the features chosen by any of the classifiers (colored nodes) are not connected to any other chosen feature. It is known that there is in many cases a correlation between expression value of the genes whose corresponding proteins interact . Therefo3As explained in Section “4What was more intriguing to see was that four out of the seven important features of the TCGA-LAML gene expression data set contained ribosomal proteins when using the risk group label, i.e. RPL37A, RPS20, RPS3A, and RPL23A. For a long time ribosomes were just considered machines that perform an unbiased translation of genes from mRNA to amino acid sequences, but this view has recently been challenged . One newMachine learning has become more and more popular in many real world scenarios for making sense of large collections of facts. Differences between the data used for training the method and new data for which the label should be predicted can limit the performance of prediction methods on those data. In this work we introduced a method that estimates these potential partial biases and incorporates them into the prediction function. We applied it to gene expression and DNA methylation measurements from cancer patients. Our method has state-of-the-art performance on many different prediction tasks. Furthermore, we show how to make sense of the predictions. Visualizing the important genes can lead to new biological insights, as shown for the TCGA-LAML data set with the risk group label. Instead of mapping the genes to PPI networks, one could also think of mapping them to signaling pathways .Recently, a study showed that most published signatures are not significantly more associated with cancer outcome than random signatures . One of In this work we designed and developed a method that besides being a predictive model, it can be used for two different purposes. It can be used as an exploratory method to reveal potential features used in future studies; and it can be used to different underlying causes of the same disease and with its interpretability help oncologists to choose the treatment accordingly.We would like to point out that the applicability of our method is not limited to cancer outcome prediction, and it can apply to many more scenarios. The method assumes that the data has enough features to select from, and that there are related features to those selected ones that can be used to estimate their reliability. These are conditions that almost all biological data satisfy, hence the method can be applied to them.github repository of the method.The method also works as a skeleton whose components can be easily substituted. For example, by changing the classifier used in individual learners to a multi-class classifier, the method would work on multi-class problems. For the sake of simplicity and without loss of generality we performed the evaluations only on binary classification problems. Also, due to the structure of our model, one possible approach would be to use a method such as iRDA and use those gene sets as features of individual learners. Whether this approach leads to better results or not requires further research. Also, the combination of maximal information coefficient and Gaussian processes is not the only feasible option, and they can be replaced with other faster methods if the time complexity of the method is of any concern. Some of these alternatives are already available on the


Sparse representation classification (SRC) is one of the most promising classification methods for supervised learning. This method can effectively exploit discriminating information by introducing a  The development of high-throughput technologies has enabled scientists to monitor the gene expression levels in tens of thousands of genes simultaneously in a single experiment. This technology has become a symbol of the post-genomic era Some dimensionality reduction methods have recently been proposed to solve the “large Sparse representation classification (SRC) was proposed by Wright et al. To capture more alternative information from gene expression data, the so-called meta-samples are proposed by The data-dependent sparsity can be automatically adjusted, rather than empirically chosen. Without computationally expensive model selection, our method is scalable and efficient.The existing MSRC Extensive experiments are performed to evaluate the proposed method. Experimental results show the superiority of the non-parametric version of MSRC compared with some state-of-the-art classifiers. Section 3 presents more details.The remainder of this paper is organized as follows: prior work on sparse representation classification and the fundamentals of the proposed method are described in Section 2. Section 3 presents the experimental results. The proposed method is discussed in Section 4. Section 5 concludes this paper.This study primarily aims to establish the manner by which to devise an robust classifier for tumor subtype classification. Given a microarray data set Gene expression profile data are high-throughput data with tens of thousands of genes. However, the number of samples is usually very small, which makes classification challenging. To avoid the curse of dimensionality, differential gene expression analysis As illustrated in Mathematically, meta-samples extraction can be regarded as a type of matrix decomposition, including non-negative matrix factorization Biologically, meta-samples are also called eigenarray Prior works revealed that meta-samples preserve some desired discriminant information of samples from the same class.In this subsection, we revisit the sparse representation problem briefly. Sparse representation is one of the most important components of machine learning and data mining community that has wide applications in such fields as text mining, image classification, and bioinformatics. In this work, we interpret the sparse representation problem from the view of linear algebra.From the standpoint of linear equations system Linear equation systems have infinitely many solutions if they are underdetermined , Eq. (4) is an unconstrained convex problem. Notably, Sparse representation assumes that a signal can be reconstructed by a small number of basis signals within a linear combination. Thus, Eq (3) can be named as basis pursuit Zheng et al. Notably, the gene expression profile contains data with high dimensionality and small sample size  method.In this subsection, we first propose a heuristic weighted strategy, the reasonableness of which is theoretically proven. We then construct an underdetermined linear equation system, in which the data-dependent sparsity can be self-adaptively tuned by Let Alternatively, Eq. (6) can be compactly reformulated as Proposition 1.Singular value is a reasonable weighting factor for measuring the importance of meta-samples.Proof. Let This completes the proof. □The evaluation metric function is used to measure the meta-sample's contribution of the meta-sample to the raw data reconstruction in terms of Let Theorem 1 proves that Eq. (8) is a underdetermined linear system. As stated in Subsection 2.2 the sparsity of underdetermined linear system can be automatically tuned by Theorem 1.Linear equation system (8) is underdetermined, and Proof. We can find a sub matrix in Note that We define a projection function Step 1. Input training sets Step 2. Normalize training set samples and testing sample to obtain unit Step 3. Extract weighted meta-samples Step 4. Solve non-parametric sparse representation problem by Eq. (8);Step 5. Compute residuals for each class Step 6. Return class label of PFMSRC can be considered as a non-parametric version of MSRC, compared with the former having the following merits:The weighted meta-samples are orthogonal with one another. That is, no redundancy exists among meta-samples, and the weight enhances the influence of the main singular vector, such that discriminant information can be well retained.The data-dependent sparsity can be automatically tuned without human intervention. Thus, PFMSRC has better scalability and robustness.The time complexity of PFMSRC is lower than that of MSRC, since computationally expensive model selection work need not be accomplished for parameter optimization. Time complexity can be estimated as: weighted meta-sample extraction step needs time complexity In the following section, we will conduct extensive experiments on micoarray data to evaluate the effectiveness of our scheme, and microarray data repository information as well as the accession number is given by In this section, we will evaluate the performance of the proposed PFMSRC algorithm against four state-of-the-art algorithms, namely, linear discriminant analysis (LDA+SVM), independent component analysis (ICA+SVM), SRC, and meta-sample sparse representation (SVD-MSRC). The former two are model based and accompanied by feature extraction. These two algorithms are regarded as baseline. For the model-based method, support vector machine Colon Acute leukemia data DLBCL Gliomas SRBCT ALL MLLLeukemia LukemiaGloub Gene expression profiling involves data with high dimensionality and small sample size. The exclusion of redundant and irrelevant data is critical for classification. As suggested by For LDA+SVM algorithm, we simply extract SRC and MSRC methods need parameter To evaluate the performance of five methods on a balanced split data set, we randomly select We show the average performance comparison on four binary classification tasks in Classification accuracy, specificity, and sensitivity are some popular evaluation metrics. In this work, we use all three to evaluate performance, and the results are reported in We investigate multiclass classification performance on four publicly available data sets. The experimental setup is the same as that for the binary classification case. On one hand from On the other hand, sparse representation based classification methods are less sensitive to the number of samples used for training model-based classification methods, which suggests a natural approach to select a classifier when the training sample size is small. In this subsection, we evaluate the performance of the five methods with different feature dimensions on eight tumor data sets. For the training data, 10 samples per subclass are randomly selected, whereas the remaining samples are used for test. We perform the test with various numbers of genes, starting from 50 to 400 genes in steps of 20. The comparison experiment was performed 20 times, and the average prediction accuracy of our experiments on eight gene expression profile datasets was recorded for evaluation.The balanced training sets for each dataset ensure fair evaluation as stated by In binary classification case, SRC, MSRC, and PFMSRC share the same curve trend. Compared with SRC and MSRC, PFMSRC performs well by using a smaller number of genes, SRC and MSRC can achieve comparable accuracy by using more genes. Evidently, SRC, MSRC, and PFMSRC consistently outperform LDA+SVM and ICA+SVM in all datasets.In the multiclass classification case, the performance of MSRC, SRC, and PFMSRC is very stable with respect to the number of genes, and all these methods converge fast to the optimal classification rate point. In To evaluate the classification performance on imbalanced split training/testing sets, we perform 10-fold stratified CV on tumor subtype dataset. All samples are randomly divided into 10 subsets based on stratified sampling: nine subsets are used for training, and the remaining samples are used for testing. This evaluation process is repeated 10 times, and the average result is presented. The 10-fold CV results are summarized in We can conclude that model-based approaches are more vulnerable to the small sample size problem, over-fitting should be resolved properly.Based on the above experiments, we can draw the following observations:Sparse representation based methods  consistently outperform the model-based methods  on all experiments. Especially, in balance splited datasets the prediction accuracy of model-based methods is significantly lower than that of sparse representation methods which may be attributed to the small sample size problem. However, SRC, MSRC, and PFMSRC perform well even when we take 5 samples per subclass for training and the rest for testing.SRC, MSRC and PFMSRC are robust to various sample sizes and feature dimensions, as well as converge fast to the optimal classification rate. The experiments verify the results in PFMSRC outperforms SRC and MSRC in most cases, which implies that the parameter free sparse representation and weighting strategies can capture more discriminating information, especially in multiclass classification. See PFMSRC is a parameter-free method, in which the data dependent sparsity can be self-adaptively tuned, compared with SRC and MSRC in which search for a regularization parameter is laborious work. Moreover, the number of meta-samples is a key parameter for MSRC, as shown in In this study, we proposed a novel non-parametric meta-sample-based sparse representation. The algorithm assumes that test samples can be well reconstructed within a linear combination of weighed meta-samples in the same class. We theoretically proved the rationality of the weighting strategy. A simple but efficient projection function is constructed by the sparse representation coefficients to complete the classification work. We also compare the performance of PFMSRC with that of two model-based methods and two sparse representation-based methods on eight tumor expression datasets. Experimental results have shown the superiority of the proposed method. We then drew some conclusions on the effects of both balanced split and imbalanced split testing/training sets on tumor classification problems.PFMSRC exhibits stable performance with respect to different training sample sizes and feature dimensions compared with the other four algorithms. Thus, the extension of the sparse representation with dimensionality reduction (feature selection or feature extraction) in a unified framework is one of our future works.


Streptococcus anginosus) seem to characterize general dysbiotic states of the microbiome rather than connections with a specific disease. Our results in modelling features of the “healthy” microbiome can be considered a first step toward defining general microbial dysbiosis. The software framework, microbiome profiles, and metadata for thousands of samples are publicly available at http://segatalab.cibio.unitn.it/tools/metaml.Shotgun metagenomic analysis of the human associated microbiome provides a rich set of microbial features for prediction and biomarker discovery in the context of human diseases and health conditions. However, the use of such high-resolution microbial features presents new challenges, and validated computational tools for learning tasks are lacking. Moreover, classification rules have scarcely been validated in independent studies, posing questions about the generality and generalization of disease-predictive models across cohorts. In this paper, we comprehensively assess approaches to metagenomics-based prediction tasks and for quantitative assessment of the strength of potential microbiome-phenotype associations. We develop a computational framework for prediction tasks using quantitative microbiome profiles, including species-level relative abundances and presence of strain-specific markers. A comprehensive meta-analysis, with particular emphasis on generalization across cohorts, was performed in a collection of 2424 publicly available metagenomic samples from eight large-scale studies. Cross-validation revealed good disease-prediction capabilities, which were in general improved by feature selection and use of strain-specific markers instead of species-level taxonomic abundance. In cross-study analysis, models transferred between studies were in some cases less accurate than models tested by within-study cross-validation. Interestingly, the addition of healthy (control) samples from other studies to training sets improved disease prediction capabilities. Some microbial species . In particular, shotgun metagenomics provide data at unprecedented species- and strain- levels of resolution. Several large-scale metagenomic disease-associated datasets are also becoming available, and disease-predictive models built on metagenomic signatures have been proposed. However, the generalization of resulting prediction models on different cohorts and diseases has not been validated. In this paper, we comprehensively assess approaches to metagenomics-based prediction tasks and for quantitative assessment of microbiome-phenotype associations. We consider 2424 samples from eight studies and six different diseases to assess the independent prediction accuracy of models built on shotgun metagenomic data and to compare strategies for practical use of the microbiome as a prediction tool. The human microbiome constitutes the whole set of microbial organisms associated with the human host. It has been shown to be crucial for human health and for the development and maintenance of the immune system and for several metabolic activities –3. SigniNext-generation DNA sequencing technologies permit comprehensive profiling of the microbial communities from human-associated samples, and have now been sufficiently widely employed to enable meta-analysis for discovering patterns common to independent studies. Meta-analysis has been broadly adopted in other genomics applications, such as for analysis of microarray or RNA-seq data, where multiple studies have been performed for a similar purpose including identifying gene expression signatures of specific human cancers. The general objectives of meta-analysis include proposing new classifiers , compariThe most common and cost-effective approach for microbiome characterization to date targets the 16S rRNA gene as taxonomic marker . Meta-anImproved resolution and lower variability of shotgun metagenomics hold the promise to provide improved generalization of microbial signatures over 16S rRNA sequencing . Meta-anIn this study we uniformly process 2424 shotgun metagenomic samples from eight studies to assess the independent prediction accuracy of models built on metagenomic data and to compare strategies for practical use of the microbiome as a prediction tool. The software framework and the microbiome profiles for thousands of samples are made publicly available.Methods). Our multi-level validation strategy includes the assessment of microbiome models on single cohorts, across stages of the same study, across different studies, and across target outcomes and conditions . The software and validation framework is publicly available at http://segatalab.cibio.unitn.it/tools/metaml and was applied on a total of 2424 publicly available metagenomic samples from eight large-scale studies (see Table 1 and Methods). All samples were processed with MetaPhlAn2 .Lasso  and ENetTwo main kinds of analysis were performed in this work, i.e., cross-validation and cross-stages/studies. For cross-validation studies, prediction accuracies were assessed by 10-fold cross validation, repeated and averaged on 20 independent runs. We underline that model selection and feature selection are done using only the training set thus avoiding overfitting problems. In the cross-stages/studies case, all the samples of the first stage/study are considered for training and thus used to generate the classification model including the model selection and feature selection steps. The generalization of the model is evaluated by applying it on the samples of the independent stage/study. In all the cases, the results obtained on the original classification problem were compared with those obtained by a random classifier (denoted in the paper as SVM-Shuffled and RF-Shuffled). For such purpose, we applied the same setting after shuffling randomly the labels of all the samples.Several different metrics were taken into account to evaluate classification performances. First, we considered the OA, which is the percentage of correctly predicted samples. From the confusion matrix three main metrics were computed: i) the precision ; ii) the recall ; iii) the F1 score, which is the harmonic mean of precision and recall, i.e., F1 = 2*/. These three metrics  can be computed for each class separately. For brevity, we report in the paper only the average values: after calculating the metrics for each class, their average values, weighted by the number of samples per class, are computed. For binary classification problems, class posterior probabilities were used to plot the ROC curve, which represents the true positive rate  against the false positive rate . From the ROC curve, we computed the widely-used AUC statistic, which can be interpreted as the probability that the classifier ranks a randomly chosen positive sample higher than a randomly chosen negative one, assuming that positive ranks higher than negative. The AUC ranges in , where 0.5 corresponds to random change.1ij and AUC2ij are the AUC of two classifiers in fold i of repetition j, and σj is the standard deviation of the AUC1ij−AUC2ij across i = 1…10 folds in repetition j. Similarly, p-values were obtained from the t-statistic obtained with mean difference and standard error smoothed over the 20 repetitions:In the comparison among classifiers, prediction accuracy was assessed by 10-fold cross-validation, repeated and averaged on 20 independent runs. The same folds were used for all classifiers, i.e. training and test sets were identical for each classifier. In this way, the difference in performance of two classifiers could be calculated directly as the difference in AUCs (or any other metric) within each test fold. Mean difference and standard error were calculated for each 10-fold CV, then averaged across the 20 repetitions for smoothing. 95% confidence intervals on the difference in AUC performance of two classifiers were calculated using the t-distribution with df = 9, i.e.:In terms of feature selection, we reported the list of the 25 most important features found by RFs. For each feature, we considered also the relative importance score, which is a real number in the range  with features that sum to 1. Feature selection is done for each run independently, and we report the average results.We initially considered a total of 2571 publicly available metagenomic samples (from eight main studies/datasets) that were reduced to 2424 after pre-processing and curation (see next sections). These are all the human-associated shotgun metagenomic studies with more than 70 samples and read length bigger than 70nt available as of January 2015. Six studies were devoted to the characterization of the human gut microbiome in presence of different diseases. Cirrhosis included 123 patients affected by liver cirrhosis and 114 healthy controls . ColorecFig 2). Before applying MetaPhlAn2 the samples were subject to standard pre-processing as described in the SOP of the Human Microbiome Project [The entire analysis was done by taking into account two types of features: species-level relative abundances and presence of strain-specific markers. These features were extracted from the metagenomic samples using MetaPhlAn2  with def Project  without The experimental evaluation can be summarized into five main steps: 1) cross-validation analysis was done on the six disease-association datasets for evaluating the capabilities of metagenomic data for disease classification; 2) cross-stage studies were performed on the cirrhosis and T2D datasets in order to test the generalization of the model on independent collection batches from the same study; 3) in terms of T2D, the analysis was extended by taking into account also samples from completely distinct cohorts; 4) cross-studies were also done to model the features of the “healthy” gut microbiome for use as a dysbiosis prediction model for syndromes where few or no training samples are available; 5) cross-validation and cross-study analysis were applied to deal with different classification problem such as gender and body site discrimination. We note that all the investigated classification problems, excluding the body site discrimination, represented binary classification problems. Moreover, most of the analysis was done in terms of disease classification, in which the objective was to discriminate between “healthy” and “diseased” subjects.-2) and “obese” (BMI ≥ 30 kg m -2) subjects for a total of 253 samples. Individuals having an intermediate BMI  were excluded. All the 344 samples of the T2D dataset were considered to discriminate between “healthy” individuals and “T2D” patients. The same classification problem was investigated in WT2D, in which 96 samples were taken into account after excluding impaired glucose tolerance women.Cross-validation analysis  in gut samples see “”. We conFinally, the developed tool was applied to non-disease-based classification problems  software is open-source, written in Python and available online at http://segatalab.cibio.unitn.it/tools/metaml together with all the data used and discussed in this work.The MetAML Click here for additional data file.S2 TableFig 5). The IntCorr is the integrative correlation [The AUC is computed by training the model on one stage of T2D and testing it on WT2D (same results in relation  between (PDF)Click here for additional data file.S1 Figp-values for the null hypothesis of no difference in AUC by 10-fold cross-validation, assessed by paired t-test, for disease discrimination on six different datasets using species abundance and marker presence as microbiome features. Comparisons that are statistically different  are highlighted in red. Average values with margins of error for AUC are reported in Fig 2.The table reports the (TIFF)Click here for additional data file.S2 FigFig 3 and S3 Fig.The corresponding prediction performances are reported in (TIF)Click here for additional data file.S3 FigSpecies abundance and marker presence were used as microbiome features. The best value for each dataset and feature type  are in bold, and the overall best values for each dataset are circled. Enet and Lasso denote their use as classifier applied on the entire set of features; RF(or SVM)-FS:Enet(or Lasso) denotes RF (or SVM) used as classifier on the reduced set of features given by Enet (or Lasso). The margins of error are reported in parenthesis.(TIF)Click here for additional data file.S4 FigMost important discriminating species (left) and markers (right) identified by RF for disease discrimination in (a) IBD, (b) obesity, (c) T2D and (d) WT2D cross-validation studies. In the left panels, for each species reported on the vertical axis, the top bar (in blue) corresponds to the feature relative importance (with standard deviation reported with error bars) and the two bottom bars refer to the average relative abundance for healthy (in green) and diseased (in red) samples. In the right panels, for each marker the top bar is coloured according to the corresponding species and the two bottom bars refer to the average marker presence.(TIF)Click here for additional data file.S5 Fig(TIF)Click here for additional data file.S6 FigFeatures and data sets were clustered using correlation similarity.(TIF)Click here for additional data file.S7 Fig(a) Cross-validation and (b) cross-study analysis for gender discrimination in multiple datasets. In bold: the best value for each scenario and feature type ; Circled: the absolute best value for each scenario.(TIF)Click here for additional data file.S8 Figa) Cross-validation results by considering one sample per subject or more than one sample per subject but each from a different body site. (b) Normalized confusion matrix when using RF on species abundance.((TIF)Click here for additional data file.S9 Figa) obesity, (b) T2D and (c) WT2D datasets are overestimated if the most relevant features  are pre-selected. Cross-validation prevents this problem.Prediction capabilities for disease discrimination in the ((TIF)Click here for additional data file.S10 Figa) Most relevant species (left) and markers (right) identified by RF by cross-validating on the entire set of samples. In the left panel, for each species reported on the vertical axis, the top bar (in blue) corresponds to the feature relative importance (with standard deviation reported with error bars) and the two bottom bars refer to the average relative abundance for healthy (in green) and diseased (in red) samples. In the right panel, for each marker the top bar is coloured according to the corresponding species and the two bottom bars refer to the average marker presence. (b) Hierarchical clustering on the feature relative importance score determined by running RF on species abundances in a "leave-one-dataset-out" cross-study validation [(lidation . For eac(TIF)Click here for additional data file.


In the last decade, a great number of methods for reconstructing gene regulatory networks from expression data have been proposed. However, very few tools and datasets allow to evaluate accurately and reproducibly those methods. Hence, we propose here a new tool, able to perform a systematic, yet fully reproducible, evaluation of transcriptional network inference methods.Our open-source and freely available Bioconductor package aggregates a large set of tools to assess the robustness of network inference algorithms against different simulators, topologies, sample sizes and noise intensities.The benchmarking framework that uses various datasets highlights the specialization of some methods toward network types and data. As a result, it is possible to identify the techniques that have broad overall performances.The online version of this article (doi:10.1186/s12859-015-0728-4) contains supplementary material, which is available to authorized users. Despite extensive knowledge of individual genes, we are still far from understanding the regulation mechanisms happening inside biological cells. In order to gain a system-level understanding, it is necessary to examine how genes interact on a large-scale level.Some specific genes called transcription factors (TF) bind to the promoter regions of target genes (TG) and can activate or inhibit a TG’s expression. Therefore, genes do not work in isolation; they are connected in highly structured networks. Gene Regulatory Networks (GRNs) represent this set of relationships.Reconstructing gene regulatory networks from expression data is a very difficult problem that has seen a continuously rising interest in the past decade, and presumably this trend will continue in the years to come due to the rich set of applications in biotechnological fields  as well as in the biomedical field . Several papers have compared and evaluated different network reconstruction methods –5. HowevHence, we propose a new extensive benchmarking framework that is fully reproducible with just one line of code and can also be easily modified to change the experimental setting or introduce a new inference algorithm. Our benchmarking strategy clearly shows that some methods perform very well on one of these artificial generators but can have poor results on another. This strongly suggests the importance of a tool that is able to test, both simply and broadly, any new proposed method. Some reviews such as  and 7]  7] evaluFor this reason, we provide a Bioconductor package that, by default, compares 10 variations of 5 datasets having more than 100 expression-measurements each. In other words, the package compares methods on 50 datasets, each with very different samples and even different amounts of noise. Using realistic artificial data allows for large number of samples that in turn, allows for reliable statistical measures indicative of performances and robustness. So far, no consortium nor database focusing on real data has assembled several thousands of homogeneous expression samples , that would allow for a similar benchmark. In this paper, we argue that a first step to support a new network inference method is to demonstrate its ability to recover regulatory networks from a broad set of realistic artificial datasets, where the truth is known and where the noise is controlled. Then, of course, a second step would be the analysis of the algorithms on real data .In this study we will show that our benchmarking strategy is highly informative for evaluating the performance and robustness of network reconstruction methods. Indeed, in this paper, we evaluate more than ten state-of-the-art reconstruction techniques using more than 50 datasets from different simulators in a high number of genes and low number of experiments scenario.With this study we found that no single method is the best across different sources of data, but at the same time this study also shows that some techniques, such as CLR , are ratIn order to provide a sound and fair comparison of the different methods, the use of various simulators is essential. A large set of gene expressions generated by various simulators is collected in what we call “Datasource” see Fig. .Fig. 1WAt this stage, the data generated by the simulators is free of noise. The noise will be added later so that it is possible to control its properties independently of the simulators and also to provide fully reproducible tests. This study involves data generated by three different GRN simulators:GNW The GNW simulator ) on the set of remaining genes . (MRNET)  method r (MRNET) , which iThe MRMR procedure returns a ranked list of features that maximize the mutual information with the target gene (maximum relevance) and, at the same time, such that the selected genes are mutually dissimilar (minimum redundancy). For every gene, the MRMR feature selection provides a score of potential connections where the higher scores should correspond to direct interactions. The indirect interactions should have a lower scores because they are redundant with the direct ones. Then, a threshold is computed as in the RELNET method.The MRNET reconstructs a network using a forward selection strategy, which leads to subset selection that is strongly conditioned by the first selected variables. The Minimum Redundancy NETworks using Backward elimination (MRNETB), uses instead a backward selection strategy followed by a sequential replacement .Genie3 The GEne Network Inference with Ensemble of trees (Genie3)  R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2015). R Foundation for Statistical Computing. [**] Bellot, P., Olsen, C., Meyer, P.E.: grndata: Synthetic Expression Data for Gene Regulatory Network Inference. (2014). R package version 1.0.0.σLocal(g);20 %, see Eq. σGlobal;10 %, see Eq In this section, we present the results of the benchmark with the presented methodology and obtained with version 1.0 of the package (see “ Availability and requirements ”). For each datasource of Table AUPR%20 ) of the total possible connections for each datasource. The table also gives the mean and variance across the 10 different datasets.Table AUPR%20  values for each datasource. Then, the best result is highlighted in bold if its metric is statistically different from the remaining values. Note that several results may be highlighted for the same datasource if they are not statistically different from each other.In order to assess the statistical significance of the results, we perform a Wilcoxon Rank sum test with Bonferroni correction  on AUPAUPR%20  values have different ranges for each datasource. Therefore, instead of aggregating AUPR%20  values, we aggregate the rank of each method, the smaller the rank the better the algorithm. Figure In order to assess the overall behavior of each technique, we need to aggregate the different performances obtained on the different datasources. But as can be seen in Table Additionally, Table Implementation in NetBenchmark package In order to generate these results we use the main function netbenchmark. In listing ?? we present the different commands used in the netbenchmark function to generate the previously presented results, note that the random seed could be used to compare a new method on the same data than those used in the present study. Results are also available at online (see Project home page in section “Availability and requirements ”) where the results of the benchmark will be updated (with most recent version of the package) with new methods or updates of the presented algorithms.κ value of σn;κ%), therefore decreasing the SNR. In this study we also use subsampled datasources of 150 experiments in order to derive the effect of noise on the various GRN reconstruction methods and being able to compare them with the results obtained at the previous study. In Table σn;κ% value, we perform ten different trials and the performance metrics (AUPR%20 ) are the average of the different trials. In Fig. Here we present a procedure in order to test the stability of the different algorithms in the presence of local Gaussian noise. To do so, we use all datasources in Table AUPR%20 ) are averaged over the different trials.The aim of this procedure is to measure the robustness of the different reconstruction methods in terms of number of available experiments. In a real world scenario, one has budgetary limitations and therefore there is a restriction on the number of different experiments that can be done. Here, we want to address this issue by identifying the best methods in several scenarios with different number of experiments. To do so, we subsample the experiments of the datasources of Table The results are presented in Table AUPR%20  across datasources: Better results can be expected for smaller networks and for simpler simulators such as Rogers. It is worth noting that PCIT and Zscore almost reach a 100 % precision over their most confident connections in the Rogers datasets , and none of the methods is the best one for all datasources. We also find large variations in terms of Differently from , we do nAUPR%20  values remain low, which highlights the difficulty of the task at hand. Still, we observe a trend of decreasing performance when the noise increases. However, we can see how the performances of ARACNE, C3NET and GeneNet are the most affected by increasing noises. The other methods appear less sensible to the noise addition.We have studied the effect of noise on the performance using an additive Gaussian noise with different noise intensities, and we have found that the majority of the methods are quite robust to the noise effects. Also, the improvement of the performance on the datasets without noise is almost negligible. Even in the absence of noise, the We also have studied the effect of the number of experiments on the performances. On one extreme, we have included a setup involving more experiments than genes and, on the other extreme, a setup where the number of experiments is around 1 % of the number of genes. We found that increasing the number of samples seems beneficial in most of the methods; it is worth noting that on datasource R1 the performance is outstanding for the Zscore, PCIT and GeneNet methods. These results are coherent with a similar study presented at . Note thAs previously stated, the present review is fully reproducible, with one function call of the Bioconductor package NetBenchmark. With this package, the different datasources are automatically loaded and the presented methods are implemented or imported with the package.R is a broadly used open source language and environment for statistical computing and graphics . NowadayWe have developed several wrappers with the default parameters for most methods. The names of the wrappers of the GRN reconstruction algorithms that are currently available in the package are listed in Table In the present study we made a set of choices such as the evaluation measure or the number of datasets per datasource, but thanks to the Bioconductor package NetBenchmark, the user can make a different sets of choices, and the package can also be used for a deeper analysis of the methods. We refer the interested reader to the help files of the package for further information.https://imatge.upc.edu/netbenchmark/) that allows following research and comparison of new methods within the same conditions.Additionally, the Bioconductor package NetBenchmark allows testing new methods with the benchmark in the same conditions that we presented in this review. The presented results are available online  a GRN reconstruction task and therefore we have taken into account the goals of the community such as the evaluation of the most confident connections. We have also developed a Bioconductor package and webpage to allow future research and comparison of new methods under the same conditions and to provide the possibility to change them. The present paper has assessed the different GRN methods in a high-heterogeneity data scenario and has highlighted the specialization of methods for the different network types and data.As a general conclusion, we can observe that CLR is the best on the majority of the datasets, but it does not obtain the best results across all the different datasources and kinds of data. In the case of complete knockout data, the best-performing methods are the Zscore followed by PCIT and GeneNet. Let us note also that Genie3 and MRNET exhibit competitive performances, however, these methods are not as fast as CLR in terms of computation time.


Previous studies have demonstrated that machine learning based molecular cancer classification using gene expression profiling (GEP) data is promising for the clinic diagnosis and treatment of cancer. Novel classification methods with high efficiency and prediction accuracy are still needed to deal with high dimensionality and small sample size of typical GEP data. Recently the sparse representation (SR) method has been successfully applied to the cancer classification. Nevertheless, its efficiency needs to be improved when analyzing large-scale GEP data.l2-norm or l1-norm of the coding residual.In this paper we present the meta-sample-based regularized robust coding classification (MRRCC), a novel effective cancer classification technique that combines the idea of meta-sample-based cluster method with regularized robust coding (RRC) method. It assumes that the coding residual and the coding coefficient are respectively independent and identically distributed. Similar to meta-sample-based SR classification (MSRC), MRRCC extracts a set of meta-samples from the training samples, and then encodes a testing sample as the sparse linear combination of these meta-samples. The representation fidelity is measured by the Extensive experiments on publicly available GEP datasets demonstrate that the proposed method is more efficient while its prediction accuracy is equivalent to existing MSRC-based methods and better than other state-of-the-art dimension reduction based methods. It haia (ALL) ,8 and aria (ALL) ,9 may noDimension reduction methods can be grouped into two categories: feature selection and feature reduction approaches. Feature selection methods , such aset al. proposed a SR-based classification (SRC) method using et al. further presented a new SR-based method for GEP-based cancer classification, termed meta-sample-based SR classification (MSRC), where a set of meta-samples are extracted from training samples, and then a testing sample is represented as the linear combination of these meta-samples by et al. proposed a new classifier, meta-sample-based robust sparse representation classifier (MRSRC) based on the MSRC method, for cancer classification [Machine learning based methods are also often called model-based methods because a predictive model is built for predicting the label of test sample. The model selection is a complex training procedure, which easily leads to over-fitting and decreased prediction performance. Recently, sparse representation (SR), a powerful data processing method that does not require model selection, has been extensively applied to face recognition ,19 and f samples . They ape method . Their efication . Their eet al. proposed a new coding model, namely regularized robust coding (RRC) for face recognition [Previous SR-based model assumes that the coding residual follows Gaussian or Laplacian distribution, which may not be effective for describing the coding residual in practical GEP datasets, and another problem is that the sparsity constraint on coding coefficients leads to the high computational cost of SRC method. To deal with the problem, Yang ognition . Here, wognition . Our extn-dimensional space. Each of the m columns consists of an n-element expression vector for a single sample. Let Let where The analysis flowchart of the meta-sample-based SR method is different from those of traditional model-based and template-based methods Figure . The cla1) The whole sample set is randomly split into two disjoint parts: training set and test set, and then the meta-samples are extracted only from the training set using singular value composition (SVD).2) The weight of each gene is calculated according to a weight function, and the genes with lower weight are removed in a test sample 3) The test sample 4) We can reconstruct the test sample for each subclass by using the meta-samples and the coding coefficient of the original test sample 5) The distance between the processed test sample and each reconstructed test sample et al. used singular value decomposition (SVD) to transform GEP data from a space of genes The meta-sample extracted from GEP data is commonly defined as a linear combination of all training samples. In this paper, a set of meta-sample is extracted from all training samples of one cancer type. We find that meta-sample can capture the structures to the data and offer biological insight. on the other hand, the linear combination of the meta-samples can roughly estimate genetic expression pattern of gene data . Alter, '' space , where twhere matrix The meta-sample set  formula where There are two key issues in solving the RRC model. The first one is how to determine the distribution where where The logistic function has the same properties as the hinge loss function in SVM , so we cwhere For cancer classification, the coding coefficients associated with the dictionary atoms from the same subclass would have big absolute values. However, we do not know which subclass the testing sample will belong to. Therefore, we actually assume that the coding coefficient where The RRC model has two vital cases when c values .When When 3C) algorithm was designed by Yang, et al. to solve the RRC model efficiently [Iteratively reweighted regularized robust coding .3C is actually a set of meta-samples which are extracted by SVD from the original training set, so we call our methods as MRRCC1 (the meta-sample-based regularized robust coding classification 1) and MRRCC2 (the meta-sample-based regularized robust coding classification 2) corresponding to the two cases RRC_L1 and RRC_L2, respectively.In literature  the RRC GEP data can be obtained by two technologies, DNA microarray and next-generation sequencing (NGS) technologies. In our experiments five microarray and four NGS cancer datasets are used to evaluate the proposed method Table . The fivhttp://cancergenome.nih.gov/). They include Breast invasive carcinoma , Kidney renal clear cell carcinoma (KIRCCancer), Lung adenocarcinoma (LUADCancer), and Thyroid carcinoma (THCACancer). All samples are matched cancer and normal tissue samples.The four NGS datasets are downloaded from the web site: The Cancer Genome Atlas (TCGA) , one of simplest classification methods, with correlation distance is used to classify cancer samples (here 5 nearest neighbors are used). For LDA method and the datasets with two subclasses, Euclidean distance is used because only one feature is extracted. To avoid over-fitting, before classification we extract only 5 features using these feature extraction methods except LDA whose number extracted is A two-stage method can be used to reduce the dimensionality of dataset before classification. The first stage is a process of adopting a gene filter method such as KWRST   or RelieExperiments indicate that the different divisions of training sets and test sets can also greatly affect the classification performance. In our experiments, the Balance Division Method (BDM) is used to divide each original dataset into balanced training sets and test sets . For thel1-norm regularization but the former has much lower computational cost [With the development of microarray and NGS technologies, a huge amount of GEP data is rapidly accumulated, demanding more efficient analysis methods to analyze these data. In this paper we present a novel meta-sample-based regularized robust coding for cancer classification (MRRCC) that firstly represents each test sample as a linear combination of all meta-samples which are extracted from the original training set using SVD. The coefficient vector is then obtained by nal cost . The expThe authors declare that they have no competing interests.Shu-Lin Wang designed the framework of analysis, performed the partial experiments, and drafted the manuscript. Liuchao Sun also performed the partial experiments, and Jianwen Fang analyzed the numerical results and revised the manuscript. All authors read and approved the final manuscript.


Comparing phenotypes of heterogeneous cell populations from multiple biological conditions is at the heart of scientific discovery based on flow cytometry (FC). When the biological signal is measured by the average expression of a biomarker, standard statistical methods require that variance be approximately stabilized in populations to be compared. Since the mean and variance of a cell population are often correlated in fluorescence-based FC measurements, a preprocessing step is needed to stabilize the within-population variances.We present a variance-stabilization algorithm, called flowVS, that removes the mean-variance correlations from cell populations identified in each fluorescence channel. flowVS transforms each channel from all samples of a data set by the inverse hyperbolic sine (asinh) transformation. For each channel, the parameters of the transformation are optimally selected by Bartlett’s likelihood-ratio test so that the populations attain homogeneous variances. The optimum parameters are then used to transform the corresponding channels in every sample. flowVS is therefore an explicit variance-stabilization method that stabilizes within-population variances in each channel by evaluating the homoskedasticity of clusters with a likelihood-ratio test.With two publicly available datasets, we show that flowVS removes the mean-variance dependence from raw FC data and makes the within-population variance relatively homogeneous. We demonstrate that alternative transformation techniques such as flowTrans, flowScape, logicle, and FCSTrans might not stabilize variance. Besides flow cytometry, flowVS can also be applied to stabilize variance in microarray data. With a publicly available data set we demonstrate that flowVS performs as well as the VSN software, a state-of-the-art approach developed for microarrays.The homogeneity of variance in cell populations across FC samples is desirable when extracting features uniformly and comparing cell populations with different levels of marker expressions. The newly developed flowVS algorithm solves the variance-stabilization problem in FC and microarrays by optimally transforming data with the help of Bartlett’s likelihood-ratio test. On two publicly available FC datasets, flowVS stabilizes within-population variances more evenly than the available transformation and normalization techniques. flowVS-based variance stabilization can help in performing comparison and alignment of phenotypically identical cell populations across different samples. flowVS and the datasets used in this paper are publicly available in Bioconductor. We describe an algorithm that transforms a collection of flow cytometry (FC) samples in order to stabilize the variance within cell populations in each fluorescence channel for the entire collection of samples. This transformation enables cell populations (clusters of cells with similar phenotypes) with homogeneous variances to be easily compared with each other by standard statistical methods. Between-population comparisons are important in detecting changes in populations across biological conditions, which might help us to diagnose diseases, develop new drugs, and understand the immune system in general –5. HenceFC technology measures morphology (from light scattering) and the expression of multiple biomarkers (from fluorescence emission of fluorophores attached to antibodies) at the single-cell level. An FC sample consists of hundreds of thousands or more of such single-cell measurements, and a study could consist of thousands of samples from different individuals at different time points under different experimental conditions , 7.Variance inhomogeneity is an inherent problem in fluorescence-based FC measurements and can be an obstacle both for manual data analysis performed by qualified cytometry operators and for automated multi-sample comparisons, which typically rely on an intermediate step of cell clustering using a plethora of approaches from modified k-means to non-parametric Bayesian methodologies , 9. The p-values may not be appropriate. First, we know that such a null hypothesis is unrealistic: biological variability, technical variability of blood or bone-marrow sample measurements, and random effects associated with the biochemistry of antibody binding will certainly produce clusters of differing means. Second, a hypothesis-testing framework addresses only the question of whether the clusters have the same location, but it is not designed to measure the magnitude of the difference or lack of homogeneity within a postulated metacluster. Finally, the p-values are affected by both cluster size and metacluster homogeneity. Thus, the p-values obtained would not be comparable for various metaclusters or different clusters within metaclusters.In order to demonstrate the flowVS results we evaluate the pre- and post-processing cluster homogeneity, and quantify the improvement offered by our approach. We report the results using a simple measure of effect size, rather than through a hypothesis-testing framework. As an example, consider the population registration problem in which corresponding cell clusters from multiple FC samples are identified based on the average levels of markers expressed by the clusters , 12–14. Variance stabilization (VS) is a process for dissociating data variability from mean signals –18. OtheThe VS problem in FC, however, cannot be solved directly by applying mature VS techniques from the microarray literature. In microarrays, each gene is measured multiple times (possibly under multiple conditions) and the between-sample variance for each gene is stabilized with respect to the average expression of the gene across samples. By contrast, variance is defined by within-population cell-to-cell variation in FC, and this within-population variance is stabilized with respect to the average expression of markers within each population. These contrasting objectives prevent us from applying VS methods from microarray literature directly to flow cytometry.z across a collection of N samples. After transforming z by asinh(z/c), where c is a normalization cofactor, flowVS identifies one-dimensional clusters (density peaks) in the transformed channel. Assume that a total of m 1-D clusters are identified from N samples with the i-th cluster having variance z/c) transformation works as a variance stabilizer if the variances of the 1-D clusters are approximately equal, i.e., We address the need for explicit VS in FC with a maximum likelihood (ML)-based method, called flowVS, which is built on top of a commonly used inverse hyperbolic sine (asinh) transformation. The choice of asinh function is motivated by its success as a variance stabilizer for microarray data , 21. flotio test . From a Using a healthy-subject data set from Purdue and publicly available immune tolerance network (ITN) data, we demonstrate that flowVS removes the mean-variance dependence from raw FC data and makes the within-population variance relatively homogeneous. We demonstrate that alternative transformation techniques might not stabilize variance. Variance homogeneity is especially useful to build metaclusters from a collection of phenotypically similar cell populations across samples , 30, 31.f. For example, z following the Poisson distribution . Then the optimization procedure works as follows: clow, chigh] into k=(chigh−clow) equal regions where the i-th region is defined by the interval  and ci+1−ci=1.(a) We divide the interval  with minimum Bartlett’s statistic. For each cofactor, we compute the Bartlett’s statistic with the steps described in Sec. i-th interval with the associated Bartlett’s statistic (b) For the c∗ as follows: (c) We identify the overall optimum cofactor Minimizing Eq. ivatives . Let cloc∗, channel z in each sample is transformed by asinh(zj/c∗) and used in subsequent analysis.Equation We demonstrate the use of flowVS and other related methods by using a healthy-subject data set from Purdue University (HD) and publicly available immune tolerance network (ITN) data. The original HD data set consists of 65 samples from five healthy individuals who donated blood on different days . Here, fWe identify lymphocytes in each sample of the HD and ITN datasets by using a two-step gating shown in Fig. At first, flowVs identifies an optimum cofactor for asinh transformation for each fluorescence channel of the HD dataset. This process is performed by identifying density peaks in each channel and minimizing Bartlett’s statistic, as described in Section We transform each sample of the HD data set by the asinh function with the variance-stabilizing cofactors and plot the density of the transformed channels in the bottom row of Fig. We stabilize variance in each channel of the ITN dataset and show the results in Fig. After identifying the optimum cofactors for each channel, we transform each sample of the ITN data set by asinh functions with the variance-stabilizing cofactors and plot the density of the transformed channels in the bottom row of Fig. http://sourceforge.net/projects/immportflock/files/FCSTrans.We compare flowVS with three automated methods developed for transforming FC data: (a) flowTrans (b) logicle (flowCore), and (c) FCSTrans. We selected these three methods because they automatically select parameters for different transformations. As discussed earlier, flowTrans estimates the parameters of different transformations  by maximizing the likelihood of data’s being generated from normal distributions . In this+ and CD4- populations separately. However, these two methods do not stabilize variances across CD4+ and CD4- populations. flowTrans fails to converge for six samples  and uses default cofactor=1 for these samples. Hence, the peaks transformed by flowTrans are on different scales, and they are hard to compare against each other. By contrast, flowVS stabilizes variance across all peaks of CD4 channels, including CD4+ and CD4- populations. Furthermore, flowVS selects a single cofactor for a channel across all samples in a data set, whereas flowTrans selects different parameters for different samples. Thus, populations are more comparable after data are transformed by flowVS.The top row of Fig. z to  scale by rescaling each element zi with the following equation (zi− min{z})/(max{z}− min{z}). For each transformation, we identify the density peaks in the converted CD4 channel and plot standard deviations of density peaks against their ranks of MFI in the bottom row of Fig. Next, we quantitatively compare the stability of variance across multiple transformation methods. This comparison, however, can not be performed on the actual transformed data because different transformations convert data to different scales. Hence, we convert each transformed channel z to 0, scale byThe comparison of different transformations on the ITN data set is shown in Fig. Bartlett’s test assumes that the cell populations are normally distributed and is sensitive to departures from normality. Density peaks (1-D cell populations) in data sets that we have studied approximately follow normal distributions. This normality assumption is typical for many FC data sets as well. Hence, a VS approach based on Bartlett’s test is expected to work well for most FC data sets. For example, in Fig. However, if cell populations deviate significantly from normality, we could use other likelihood ratio statistic that is less sensitive to departures from normality, such as Levene’s  or the B+ cell clusters from the first and the second samples of the ITN data set, we observe p-values less than 10−10 for all transformations despite the fact that the tested cell populations are biologically identical (in an immunophenotypic sense).We now briefly demonstrate the impact of variance stabilization on the homogeneity of metaclusters . In metacluster homogeneity evaluation, the underlying assumption is that all clusters in a metacluster represent the same cellular immunophenotype. As mentioned before, the hypothesis-testing framework may not be appropriate for the described problem, since a null hypothesis claiming that all clusters in a metacluster have equal mean is essentially always false. Moreover, when the number of cells (sampling units of the test) increases, the power of a statistical test such as a t-test or an F-test increases too. Consequently, a statistical test would inevitably detect small  differences between clusters. For example, performing a t-test with CD4p-values to illustrate the impact of the proposed algorithm on metacluster homogeneity. We employ the ratio of between-cluster variation  . Conside+ metacluster and a biologically erroneous metacluster grouping CD4+ and CD4- cells. The values are computed after the data are transformed by four transformations considered in this paper. In this example, the flowVS transformation gives the best homogeneity within the CD4+ metacluster, and an increased heterogeneity of the CD4+/CD4- clusters mixture. Thus application of flowVS not only results in the highest homogeneity of a set of known phenotypically identical cell clusters, but also provides the best discrimination between homogeneous and heterogeneous collection of clusters. The result demonstrates that the flowVS-based variance stabilization can help in performing comparison and alignment of phenotypically identical cell populations across different samples.Unlike the F-test for comparing multiple clusters, the above ratio does not depend on the sample size, and it is constructed so that the increasing homogeneity of a metacluster results in a progressively smaller value of the ratio. Table m genes are measured from N samples in a microarray experiment. After transforming the data by the asinh function, the mean μi and variance ith gene gi are computed from the expressions of gi in all samples. flowVS then stabilizes the variances of the genes by transforming data using the asinh function with an optimum choice of cofactor. Unlike FC, a single cofactor is selected for all genes in microarrays.The VS approach based on optimizing Bartlett’s statistic can also be used to stabilize variance in microarray data. However, the initial steps of flowVS need to be adapted for microarrays. Assume that the expression of We have applied the modified flowVS to the publicly available kidney microarray data provided by Huber et al. . The kidWe compare the VS performance of flowVS with two software packages, VSN by Huber et al.  and DDHFIn Subfig. x-axis and thus make it easy to visualize the homogeneity of variances. We also show the running median estimator of standard deviation by the red lines. Both VSN and flowVS remove the mean-variance dependence because the red lines are approximately horizontal for both transformations. Hence, flowVS performs at least as well as a state-of-the-art approach developed for microarray data.To take a closer look at the transformed data by flowVS and VSN, we plot the variances of genes against the ranks of their means in Fig. We describe a variance-stabilization framework, flowVS, that removes the mean-variance correlations observed in cell populations from FC samples. This framework transforms each fluorescence channel by the asinh function whose normalization cofactor is optimally selected by Bartlett’s likelihood-ratio test. Variance homogeneity (homoskedasticity) is a desirable property for comparing populations across conditions, building metaclusters from phenotypically similar populations, and analyzing metaclusters in an ANOVA model. However, unlike the earlier approach by Hahne et al. , flowVS flowVS is built on several assumptions that limit our approach. First, flowVS stabilizes variance separately in each channel. Thus it might be unable to stabilize covariances across multiple channels when they are correlated. Second, flowVS identifies 1-D density peaks and evaluates the homogeneity of populations by the likelihood-ratio test. Therefore, this algorithm might not perform well when density peaks are not easily identifiable. Third, flowVS stabilizes variance more accurately when a number of samples are simultaneously passed to the algorithm. Hence, this approach is not suitable for normalizing a single sample or stabilizing variances of sequentially arriving samples. Finally, Bartlett’s test used in flowVS assumes that the deviation from normality is relatively modest. If data deviate significantly from normality, other likelihood ratio tests can be employed, such as Levene’s test  or the BflowVS operates as an independent module in the FC data analysis pipeline. It does not depend on the preprocessing algorithms applied before VS nor on the post-analysis methods such as matching, metaclustering, and classification. Hence, flowVs is capable of working with most automated clustering and metaclustering algorithms developed for flow cytometry.asinh, inverse hyperbolic sine; FC, flow cytometry; ITN, immune tolerance network; MFI, mean fluorescence intensity; VS, variance stabilization.


After RNA extraction, reverse transcription and qPCR were performed. Samples were divided into four analyzed groups. Fifteen candidate RGs were tested by RefFinder tool and manual statistics. To present the importance of RG selection, TP53 gene expression levels in samples were normalized with the use of RG data. RPL13 gene was the most stable RG in analysis of 35 primary tumor nonmetastatic versus 35 mccRCC samples and matched metastasized T/C/M samples . GUSB was the most suitable RG in total 152 samples and in paired T and C (n = 140) kidney samples. Expression of GUSB, RPL13, and the RPL13 + RPLP0 pair were independent of clinical/sample variables. Normalization of TP53 expression levels showed variability of GAPDH and ACTB assays. GUSB or RPL13 assays should be used in mccRCC for qPCR data normalization whereas GAPDH and ACTB assays should be avoided. Prior RG studies should precede each qPCR gene expression study since RG selection is associated with the origin and proportion of specimens.There is no data on reference gene (RG) selection in metastatic clear-cell renal cell carcinoma (mccRCC) for quantitative PCR (qPCR) data normalization. We aimed at selecting the most stable RG for further determination of new prognostic markers. Thirty-five nonmetastatic and 35 mccRCC patients undergoing radical nephrectomy were included. Paired primary tumor  contains supplementary material, which is available to authorized users. Renal cell carcinoma (RCC) is the third most common genitourinary malignancy, and its incidence has increased in the last 20 years . The mosGAPDH or ACTB genes [Most ccRCC gene expression studies are normalized to TB genes , whose vTB genes –10. TherTP53 gene expression rate with the use of obtained normalization data of all RGs in order to show that the gene expression results in ccRCC strongly depended on RG selection. The results of such molecular approach have not been published yet.The second aim of the study was to analyze n = 2), brain (n = 2), brain and liver (one patient)  due to additional lymphadenectomy and/or adrenectomy performed during nephrectomy , Poland, between January 2011 and May 2013. The use of tissue material was approved by the Medical Ethical Committee of the MUG (decision no. NKEBN/4/2011), and informed written consent regarding the use of tissue was obtained before surgery from each ccRCC patient. One hundred fifty-two samples were classified into four groups as shown in Fig. The dissected tissue samples of primary ccRCC tumor, normal kidney, and adrenal gland (ca. 7 ± 2 mm × 7 ± 2 mm × 7 ± 2 mm) or the whole lymph node (ca. 10 mm × 10 mm × 10 mm) were collected in the operating room no longer than 20 min after the kidney resection and placed in approximately five volumes of RNAlater . Three sectioned pieces of each sample were made. The central piece was used for RNA extraction, while the two side pieces were fixed in formalin and embedded in paraffin, followed by H&E staining and the examination performed by pathologist.Total RNA isolation was performed using GeneMATRIX Universal RNA Purification Kit . Briefly, the tissues were homogenized in 2-ml tubes with ceramic beads  in the presence of 300 μl lysis buffer (Eurx) in the MagnaLyser apparatus  for 45 s at 6,000 rpm. Further processing was performed following the manufacturer’s (Eurx) protocol. Isolated RNA was eluted with 70 μl of nuclease-free water (Eurx), followed by quantification with spectrophotometer . The RNA integrity and quality were characterized by RNA integrity number (RIN) with the RNA 6000 Nano Kit using the Eukaryote Total RNA Nano Chip and Bioanalyzer 2100 apparatus . Next, 20 μl of extracted RNA was treated with TURBO DNA-free kit (Ambion) according to manufacturer’s protocol.18 primers , 200 U RevertAid Reverse Transcriptase, 1 mM dNTP mix, and 2 U Ribo-Lock . RT reaction was performed according to manufacturer’s protocol, and the resulting cDNA was stored at −25 °C after 10× dilution with nuclease-free water to be used as the template in qPCR analysis.Complementary DNAs (cDNAs) were polymerized from 2 μg total RNA (100 ng RNA/1 μl RT reaction) of each sample using 0.5 μg oligo(dT)The primers were designed using Primer-BLAST software. The calibration curves for all gene-specific qPCR assays were performed (data not shown), and the resulting calibration curves’ data are presented in Table The selection of RG assays for this study was based on the following: MeSH database search for the most commonly used RGs in ccRCC and in other cancers; previous literature results of normalization studies of kidney and other cancers , 10–14 aFor the RGs assays, the 15-μl reaction mixture included 1.5 μl 10× diluted sample cDNA, 0.2 μM each forward and reverse primers and SensiFast Sybr™ No-Rox kit . qPCR reactions for all RGs were performed in triplicate in StepOne Plus apparatus , and the geometric mean of cycle threshold (Ct) values was used in further analyses with StepOne Software ver. 2.2 (Life Technologies/Applied Biosystems). We set runs for each gene assay separately. Each qPCR run contained paired T/C or T/C/M samples, no-template control (water instead of cDNA), and 10× diluted pooled cDNA as a run-to-run precision control.GUSB assay in 15 replicates during the same run. Inter-batch (run-to-run) test was based on data obtained from separate runs (n = 5) as described above, for each RG assay separately. In both tests, raw Ct data were collected and mean ± standard deviation (SD) values were computed. Ct coefficient of variation (CtCV%) values were calculated using the following formula: CtCV = SD/mean × 100 %.Intra-run and inter-run precision control tests were set using 10× diluted pooled cDNA in the same conditions as routine RGs assays. The intra-batch control was based on qPCR reactions for TP53 which was targeted for normalization in ccRCC was amplified using the same conditions as RGs (details in Table The gene Immunohistochemistry (IHC) methodology for TP53 protein is presented in U (A) and Kruskal-Wallis ANOVA tests (B) were used to compare clinical and qPCR data.Raw Ct data was transferred and calculated using GraphPad Prism, ver. 6.05 . Normality of qPCR and clinical data was checked using D’ Agostino and Pearson omnibus test. Nonparametric Mann-Whitney The free online software tool, RefFinder  was applGeNorm data for a pair of RG was transferred from RefFinder results. For NormFinder RG pair selection, raw Ct data were calculated into arbitrary units (AU) with the use of calibration curves data Table .Raw Ct qPCR data of RG assays were compared with different clinical variables Table  with theRaw Ct data for each RG assay were divided according to the sample origin  and analyzed group (I–IV); statistical association between samples’ Ct values were assessed using A or B tests.TP53 was calculated using raw Ct data normalized with the use of Ct data of each RG. Further, expression data of TP53 in control samples was set to 1 and the expression of TP53 in T and M samples were calibrated to results obtained in control samples.The expression of 260/280 ratio was 2.02 ± 0.07 (range 1.96–2.13); average RIN values from all samples were 8.32 ± 1.12 (range 5.5–9.8).The average RNA concentration was 379.51 ± 222.21 ng/μl . All samples presented high RNA quality and integrity; mean AGUSB assay . Obtained results fulfill the precision criteria of bioanalytical method validation (CtCV% < 15 %) [As a part of a validation procedure of the presented qPCR method –21, pool < 15 %) , 22.Raw Ct data of all RGs assays ; RG assay with the smallest SD number is judged as the most stable. GUSB was the most stable RG in group I , GAPDH in group II (paired 70 T + 70 C), followed by RPL13 in groups III (35 nonmetastatic vs 35 mccRCC) and IV (matched tumor-metastasized control group of samples).dCt method was applied by Silver et al.  and is bGUSB as the most stable in groups I and II, whereas RPL13 and RPL32 were selected as best RGs in groups III and IV, respectively  based on a geometric mean of Ct values of at least two RGs. Stepwise exclusion of the least stable gene allowed the genes to be ranked according to the stability value  [M values, and the results are shown in Fig. GeNorm calculates a normalization factor (ability) . RefFindGUSB was selected as the best RG in groups I and II, and RPL13 was the most suitable gene in groups III and IV.RefFinder, free online tool, combines results of the mentioned single-RG algorithms. Besides RefFinder’s availability and simplicity (copy-paste of raw Ct data staged in columns), the more RGs are tested, the more precise result is obtained. Figure GAPDH + PPIA was selected by GeNorm as best pair in groups I and II). As a result we selected seven different RG pairs in four sample groups. In order to clarify this disagreement, we found that the CtCV% factor has been recommended to validate NormFinder and GeNorm results [GUSB vs 8.21 and 8.36 % for NormFinder’s pair for groups I and II, respectively, Supplementary Table RPL13 presents CtCV% value between pairs of NormFinder and GeNorm. Based on those results, we suppose that despite RG pairs selected by NormFinder better meet the statistical requirement than GeNorm’s, they do not outdistance single RG selection.Due to recommendation to use at least two normalizers for gene expression analysis instead of one , 20, GeN results . We checP > 0.05 between subgroups) can be finally chosen as a suitable RG in selected group of samples.Apart from selection of the most stable gene in selected groups with the use of specialized tools, the final result of RG selection has to be supported by manual statistics which utilize clinical variables and material division into subgroups according to their origin , 10, 14.GAPDH, GUSB, IPO8, RPL13, RPL32, and UBC genes, as well as RPLP0 + TBP, RPL13 + RPL32, RPL13 + RPLP0, and ACTB + TBP RG pairs, were the only assays whose expression did not depend (P > 0.05) on cancer progression. Since the expression levels of remaining RGs were associated with at least one selected clinical factor, they should not be further included in normalization panel of ccRCC studies  expression pattern between subgroups; if 70 T versus 70 C (group II) were assessed, expression pattern of six genes was comparable  in comparison to group I with only four genes when 12 M subgroup was introduced. We therefore observed the highest similarity between RG expression levels between samples when nonmetastatic versus metastatic tumor specimens were compared . Therefore, only GUSB, IPO8, PPIA, and RPL13 genes, as well as PPIA + RPL13, RPL13 + RPLP0, and ACTB + TBP pairs, passed this test. The different levels of TP53 expression for ACTB, GAPDH, and B2M in relation to number of analyzed cases should exclude those genes from RG panel for ccRCC, since the expression results of TP53 cannot change by the increased number of analyzed samples . We also observed only small increase of TP53 expression (ca. 2× increase vs control) in metastasized samples when GAPDH was utilized for normalization in contrast to 10× higher expression when RPL13 was used as RG  with the use of immunohistochemistry IHC, . IHC staRNA Fig.  in metasTP53 normalization approach, we found that GUSB is the most suitable single RG for groups I and II and RPL13 is best RG for groups III and IV  together with exclusion results of clinical/biological versus RG expression statistical tests, as well as IV Table . DiffereIn this study, we presented methods and results of selection of suitable RGs in metastatic ccRCC, because currently, there is no data concerning this important issue. A reliable normalization project should be based on validation procedures related to analytical biochemistry , 19. SinTo date, only three groups performed similar RG selection studies on tumor ccRCC compared to normal kidney tissue , 26, 27,TBP + PPIA as the most suitable RG pair, their study on 25 cases introduced 24/1 male/female (M/F) and no patients with diagnosed metastasis, whereas our project contained larger population sample (41/29 M/F) and 35 mccRCC (T3 or N1-2 or M1) cases. The most recent report on RG selection in ccRCC shows that PPIA and RPS13 (or their combination) is based on paired tumor control kidney specimens (no metastasized tissue) from 16 (14/2 M/F) ccRCC patients [GUSB or RPL13 genes [Our results of RG selection are not in agreement with other authors who focused on RCC. Although Jung et al. found patients . Furtherpatients . Finally13 genes .The limitation of our study is, however, the lack of distant metastasized samples  and utilization of only 12 samples obtained from regional metastases. Therefore, our findings should be confirmed by independent studies performed by other teams.Our results also contained the selection of RGs pair. Although the use of two or more RGs in gene expression studies has been strongly recommended by minimum information for publication of quantitative real-time PCR experiments (MIQE) guidelines , cancer We chose to create different analyzed groups of samples and presented complex results based on this division. It was connected to the fact that we utilized a large number of specimens; therefore, we decided to present as much useful data as possible. Although the presentation of more than only one comparison (nonmetastatic vs metastatic tumor samples)  could beTP53 gene expression has clearly shown that the choice of an appropriate reference gene can strongly affect final results of gene expression studies. We chose the TP53 tumor suppressor gene, whose decreased expression in tumor samples of ccRCC and other malignancies was proven by other studies [GAPDH and ACTB from the normalization panel, since false results can be obtained when normalized to those RGs. Although GAPDH is the most common RG in RCC studies and most author may disagree with our results, GAPDH expression stability should also be questioned when the biological characteristics of RCC is taken into consideration. Since tumor RCC is characterized by dysregulation of glucose metabolism [GAPDH [TP53 normalization in metastasized group with the use of GAPDH versus RPL13 reinforced the variability of GAPDH, since we confirmed strong overexpression of TP53 in metastasized tissues  by IHC, as well as other authors found its increase expression in this type of samples [TP53 mRNA content in cells instead of protein and confirm the results obtained by qPCR method. Since this method is currently unavailable to our team , we aimGUSB was noted as a suitable RG in lung [RPL13 was the best RG in Alzheimer’s disease [RPL13 in groups III–IV and may be caused by the high proportion of poorly developed  cells of tumor specimens in analyzed groups, which can be characterized by the increase deregulation or cellular machinery as compared to other specimens. Furthermore, the high differences in RG results between groups I–II and III–IV may be connected to the high proportion of metastatic tumor samples—from one fourth in group II to one half in group III, as well as to the introduction of metastatic tissue samples in group IV. We wanted to emphasize this factor, because not only the different type but also the proportion between the numbers of samples of different origin have strong influence on RG selection.In comparison to GAPDH, GUSB and RPL13 proteins do not seem to be involved in ccRCC development, since GUSB is not engaged in intracellular metabolism, whereas RPL13 is highly expressed in the cytoplasm of all nucleated cells. Due to high stability,  in lung  and ovar in lung  cancers, disease  and in m disease . We suppGUSB or RPL13) and to avoid the following RGs for normalization: ACTB, B2M, GAPDH, TFRC, and UBC.For gene expression studies of mccRCC, we recommend to use a single RG (ESM1(DOCX 12 kb).Supplementary Table 1Expression stability ranking order of candidate reference genes divided into four groups (DOCX 21 kb).Supplementary Table 2Ct coefficient of variation percentage values for each gene or genes’ pair in groups’ analyzes (DOCX 16 kb).Supplementary Figure 1GUSB assay. Amplification plot (left) shows fluorescence readings of 15 samples containing 10× diluted pooled cDNA from all ccRCC samples and 2 negative controls (green lines). Blue line—threshold , Y axis – log fluorescence, X axis – cycle number. Right plot – melting curve of samples showing one specific PCR product. No amplification in negative controls – green lines close to X axis. Plots created in StepOne Software, ver. 2.2 (PPTX 967 kb).qPCR intra-run precision control experiment for Supplementary Figure 2Summary expression data of each potential reference gene. Box  and whisker  plot of qPCR expression results (Y axis) for candidate reference genes (X axis) in alphabetical order. Shaded boxes—assays which passed D’ Agostino normality test. (GIF 26 kb)High Resolution Image (TIFF 234 kb)Below is the link to the electronic supplementary material.