Cross-validation (CV) is a technique to assess the generalizability of a model to unseen data. This technique relies on assumptions that may not be satisfied when studying genomics datasets. For example, random CV (RCV) assumes that a randomly selected set of samples, the test set, well represents unseen data. This assumption doesn’t hold true where samples are obtained from different experimental conditions, and the goal is to learn regulatory relationships among the genes that generalize beyond the observed conditions. In this study, we investigated how the CV procedure affects the assessment of supervised learning methods used to learn gene regulatory networks (or in other applications). We compared the performance of a regression-based method for gene expression prediction estimated using RCV with that estimated using a clustering-based CV (CCV) procedure. Our analysis illustrates that RCV can produce over-optimistic estimates of the model’s generalizability compared to CCV. Next, we defined the ‘distinctness’ of test set from training set and showed that this measure is predictive of performance of the regression method. Finally, we introduced a simulated annealing method to construct partitions with gradually increasing distinctness and showed that performance of different gene expression prediction methods can be better evaluated using this method. This approach typically creates an over-parameterized model-fitting problem, since any subset of the hundreds to thousands of TFs coded in the genome may be regulators of a gene, and without prior knowledge of these regulators, the relationship between every TF and putative target gene must be encoded as a free parameter in the model. Different methods adopt various strategies to handle the resulting ‘curse of dimensionality’, and since a ‘ground truth’ GRN is almost never available to allow for an unbiased assessment of these methods, studies usually apply K-fold cross-validation (CV) for evaluation6. This involves dividing the samples – experimental conditions under which expression of a gene  is known – into K disjoint subsets or ‘folds’, training the model’s free parameters on K-1 of these folds, using the model to predict gene expression in the left-out samples, and finally comparing the predictions with measured values  are used as a test sample once. Finally, by averaging the accuracy of predictions across all K repetitions, an overall measure of performance is obtained.A common method for computational reconstruction of gene regulatory networks (GRNs) is to build ‘expression-to-expression’ models, which predict gene expression as a function of expression levels of other genes, or of transcription factors (TFs) in particular Fig. ; regulatues Fig. . This is9 do not adopt this evaluation strategy, and instead of learning a global GRN, a GRN per context is learned. In this study, our focus is only on methods for inference of global GRNs using transcriptomic data and CV-based methods for their evaluation. Cross-validation is a widely accepted approach for assessment of prediction (classification or regression) methods. Among the advantages of using CV is the fact that accuracy of two different models can be compared even if the models are of different complexity, e.g., if one model employs more TFs to explain the gene’s expression than another model. It is also an extremely valuable tool in bioinformatics, since experimental testing of predictions is often tedious and expensive and is typically done on a limited scale.The reasoning behind using CV to (indirectly) evaluate the accuracy of GRN reconstruction methods is that the relationships between TFs and their target genes remain mostly unchanged across conditions, and therefore a model correctly identifying TF-gene relationships based on the training set should make good predictions on the test set. We note that studies of context- or condition-specific regulatory relationships10, and does not necessarily imply an accurate encapsulation of the input-output relationship in the trained model. . In the context of GRN recovery, in which a data sample may be a single experimental condition represented by a vector of TF expression values, less extreme versions of the above scenario may manifest  a test sample is likely to have a copy of itself present in the training set. A supervised learning method may predict accurately on such a sample since it has already seen that sample during training. In this case, the high predictive accuracy of the model in the CV evaluation may simply be the result of this proximity between training and test points est Fig.  through est Fig.  – may be12. Indeed, some previous studies have recognized this issue, in contexts related to GRNs and otherwise, requiring CV to be followed by ‘cross-condition’ assessments such as testing of models on data from unseen cell types14.One might legitimately argue against the above critique: a supervised prediction method is only expected to learn how to predict on unseen samples that are drawn from the same distribution as training samples; an evaluation of its performance ought to respect this assumption, as is the case with CV on random partitions. We do not question this point of view in the general case, but only in contexts in which the model evaluated using CV is intended to elucidate feature-response relationships that are supposed to be generalizable beyond the characteristics of the dataset on which it was trained. For example, in global GRN reconstruction, training a prediction model is only a means to learn TF-gene regulatory relationships, which are represented by the parameters of the model; the actual prediction on unseen data is only indirectly relevant. If the CV is set up in a way resembling the above scenario, with test samples being similar to  training samples, then a model may perform well on test data without accurately learning TF-gene regulatory relationships, yielding strong CV performance but not so strong GRN inference, as was observed in17 – and illustrate that CCV provides a more realistic estimate of performance on unseen samples compared to random cross-validation (RCV), the standard method of CV.  In the CCV approach, we create CV partitions by first clustering the experimental conditions and including entire clusters of similar conditions as one CV fold. This strategy may be thought of as defining one group of trans-regulatory contexts as training data and testing a regression method trained on these contexts for its ability to predict gene expression in an entirely new regulatory context.In this study, we seek to explore the aforementioned problem and to identify and assess alternative strategies so as to avoid the pitfall mentioned above. More specifically, our goal in this study is not to propose a new GRN recovery algorithm, but to propose new CV strategies better capable of assessing the generalizability of such algorithms on unseen data. To this end, we first use clustering based cross-validation (CCV) – a non-standard approach to CV previously proposed in other contextsNoting that the main difference between CCV and RCV is the degree to which CV folds are distinct from each other, we next propose a score for the ‘distinctness’ of a test experimental condition (sample), based on a given set of training conditions. This score is independent of the model/algorithm that might be used for prediction. In addition, it is computed without the knowledge of the target gene’s expression levels in either training or test conditions, and is purely based on the predictor variables . We then generalize this to define a distinctness score for any given partition of the samples into training and test sets and show that gene expression prediction accuracy is highly (negatively) correlated with the distinctness score, confirming our expectation that it is easier to predict gene expression in a test condition that is similar to training conditions.18 on real transcriptomic data using RCV and CCV, we find that the RCV creates partitions for which the test folds are relatively easily predictable. On the other hand, relying on CCV for evaluation imposes several constraints. First, CCV inevitably inherits characteristics of the utilized clustering algorithm, such as dependency on the number of clusters, initialization, etc., and introduces the new problem of how to choose the clustering algorithm and its parameters, thereby adding subjectivity to the CV process. Second, a poor choice of clustering parameters may result in CCV not generating sufficiently distinct test/training partitions, thus failing to address the above-mentioned problem. To address these issues, we propose a CV approach based on simulated annealing (SACV) to construct any desired number of partitions spanning a spectrum of distinctness scores, allowing for a more controlled use of cross-validation for assessment and mutual comparisons of different methods of expression-to-expression modeling. Using this approach, we show that while the performance of two prediction algorithms (support vector regression (SVR) and Elastic Net) for expression modeling can be very similar for partitions with low distinctness score , the difference between their performances becomes evident as the distinctness of partitions increases.By comparing the performance of a LARS (Least Angle Regression) algorithmOur new CV strategy allows us to compare two algorithms not simply based on two numbers , but rather based on many such values obtained from a spectrum of partitions with different distinctness scores. This strategy makes the test/training distinctness an explicit and well-defined parameter of the evaluation process and enables us to evaluate the performance of different methods while considering the heterogeneity between the training set and the unseen dataset for which the method is intended. For instance, a method that performs better for small distinctness values may be more appropriate in cases where the unseen dataset is expected to be similar to the training set, while a method performing better for large distinctness values is appropriate when these two sets are expected to be dissimilar. Note that this feature of SACV has applications beyond the GRN recovery problem, as it can provide new insights on the performance of any classification/regression algorithm.E of (1) all TFs T and of (2) a gene of interest g, in a set of conditions C; from these data we must learn how to predict the expression of gene g from the expression of all TFs, in any condition, seen or unseen. (We assume here that g ∉ T.) We denote the expression of a TF t in condition c by Ect and that of a gene g by Ecg. The vector of all TF expression levels in condition c, i.e., Ec. The gene expression prediction problem may thus be considered as a multivariable regression problem, where a model is trained to map the input vector Ec to the scalar value Ecg.In a common formulation of the gene expression prediction problem, the goal is to predict a gene’s expression in a cellular condition, given the expression levels of all possible regulators (TFs) in that condition. We are given the expression levels Ecg from Ec even for c ∉ C, i.e., it must generalize well to predict accurately on unseen data; as such, model training must avoid over-fitting. Often, the number of conditions with available data (|C|) is in the tens or hundreds, and the number of TFs (|T|) is in the hundreds or thousands, i.e., the dimensionality of input vectors is of the same or greater order of magnitude as the number of training samples, making over-fitting a serious concern. This also means that the model evaluation has to be performed in a cross-validation setting, which may be loosely described as follows: assuming expression data are available for a set of conditions C, choose a subset of conditions R ⊆ C, called the ‘training conditions’, train the model using Ec for condition c ∈ C–R, called the ‘test condition’, and compare its prediction C into sets R and C – R (the training and test conditions) a ‘CV partition’. A ‘K-fold CV partition collection’ is defined as follows: all conditions in C are divided into K mutually exclusive and exhaustive subsets or ‘folds’ S1, S2, … SK ; each subset Si is then used to define a CV partition by treating Si as the set of test conditions and Ri = C – Si as the training conditions. The collection of these K partitions is called a ‘K-fold CV partition collection’.The model must be able to accurately predict acy Fig. . We call19 (see Methods). These served as the dataset for analyzing various cross-validation approaches. In addition, we obtained PanCan-normalized gene expression data from four other cancers from TCGA, to serve as ‘held-out’ data. Moreover, we obtained a list of human TFs from the Animal Transcription Factor Database 20, of which 1239 were present in this TCGA dataset. Since this dataset contains transcriptomic data from different biological contexts , it allows us to explore the effect of different partitioning methods on the generalizability of the learned model to a new biological context.In order to build and compare different expression-to-expression models and assess different evaluation strategies, we obtained mRNA expression data from The Cancer Genome Atlas (TCGA) corresponding to 12 cancers in the PanCan12 dataset21 to map these samples onto a two dimensional space  techniqueet al.6 we used the LARS regression method18 as the gene expression prediction algorithm in our study. Our goal was to model a gene’s expression in a sample based on the expression of 1239 TFs in that sample. To this end, we selected 500 target genes, which had the highest expression variance across the 864 PanCan samples. Following Chandrasekaran C into six equal folds, and defined each CV partition as a training set R of conditions in five of the six folds (720 samples) and a test set S = C–R comprising the remaining conditions. We repeated this procedure 10 times, obtaining 10 different 6-fold RCV partition collections. As was expected, RCV ignored the underlying distribution of the data and placed training and test samples close to each other. This is shown in Fig. To evaluate the prediction performance, we performed cross-validation using two approaches: RCV and CCV. In RCV, we randomly divided all conditions x folds 70 samplesWe evaluated the accuracy of LARS-based gene expression prediction for each of the 20 CV partition collections described above. Accuracy was measured using Pearson correlation (higher is better) between the predicted and the measured expression values of the 864 samples. Note that the predicted value of a sample is the value obtained using LARS when that sample is in the test set. Since each of the 500 genes represents a separate prediction task, we obtained a Pearson correlation coefficient (PCC) for each of these genes. Figure Our first analysis used a balanced design where the sample size was the same for each cancer type. To test if our result held true in more realistic unbalanced cases, where the sample sizes varied by cancer type, we repeated the analysis using a different subset of the PanCan12 dataset, henceforth called the “biased dataset”. The biased dataset contains the same total number of samples as our first dataset , but instead of choosing an equal number of samples per cancer type , we set the representation of each cancer type to be proportional to its presence in the PanCan12 dataset (see Supplementary Results for more details). As shown in Supplementary Figs Given these two contradictory assessments of the LARS prediction performance , we studied which one of the two CV strategies provides a more accurate estimate of the LARS prediction performance when a new set of samples corresponding to entirely different biological contexts is used as the test data. To this end, we formed four new test sets, each corresponding to one of the four different cancer types  that were not included in the PanCan12 dataset above. The 864 samples from the PanCan12 dataset were used as the training set. Figure Although CCV provides a more realistic estimation of accuracy compared to RCV on datasets dissimilar to the training set, it relies on a clustering algorithm and is limited by the latter’s properties. Choice of the number of clusters, the clustering criteria driving the algorithm, or even its initialization may severely affect the algorithm’s output, and hence the performance estimate obtained using CCV. As a result, CCV introduces the new problem of how to choose the clustering algorithm and its parameters. Moreover, the clustering algorithm used in CCV may not generate sufficiently distinct test/training sets. When it does produce distinct sets, the clusters may be at different distances from each other and averaging the estimated performance over all clusters may not be a reliable estimate of performance on new samples.To address the above issues, we first sought to quantify how distinct training and test sets are from each other. Doing so should allow us to report cross-validation accuracy under controlled levels of training/test set distinctness, e.g., by selecting as test sets those clusters that are distinct from the remaining clusters (used as training sets), and discarding those partitions that lead to relatively similar training and test sets. Explicitly quantifying the distinctness of training and test sets may also allow us to directly construct test/training sets of a specified distinctness, bypassing the clustering step altogether, as discussed later in the manuscript.i (represented by its TF expression vector Ei) a score η capturing its distinctness from the training set R:D is the min-max normalized Euclidian distance (see Methods) between TF expression vectors Ei and Ej, representing the test condition i and training condition j, respectively. Our choice of the harmonic mean in equation  partition is negatively correlated with the performance of the trained regression model. To test whether the distinctness score we defined above satisfies this condition, we evaluated the distinctness of the 10 RCV and the 10 CCV partition collections (described above), and compared the distinctness values to the PCC measures of accuracy. Figure 22 to sample partitions of gradually increasing distinctness score (see Methods for details). In each step, this algorithm generates a new CV partition with a higher distinctness score. Figure As we previously demonstrated Fig. , test anFigure We noted above Fig.  that CV To test for the above possibility, we used the previously constructed simulated annealing based CV partitions, which comprised of 30 partitions each containing 200 test samples and 664 training samples.  We evaluated three different expression prediction methods for their ability to predict the expression of the 500 selected genes, using these 30 CV partitions . In such cases, one needs to evaluate the performance generalizability of gene expression prediction methods to new samples that may not be well represented by the training samples. As we showed in this study, random cross-validation provides an over-optimistic performance evaluation, which does not generalize to new biological contexts. This is due to the fact that RCV may place samples with high similarity in both the training and test sets, blurring the line between seen and unseen samples. This may happen to different extents in different datasets, which makes it difficult to interpret, in any absolute sense, the performance estimated by CV on different datasets: two studies may report the same CV prediction accuracies, but depending on the heterogeneity of their datasets the reported accuracy in one study may be more generalizable than the other.To address these issues, we first proposed a clustering-based CV, which allows us to obtain a more realistic estimation of error on distinct test samples. On the other hand, since CCV relies on a clustering algorithm, the choice of the algorithm and its parameters may play an important role on the obtained clusters and hence the performance evaluation. In addition, the similarity of the unseen data to the training set is a significant factor in determining which one of RCV or CCV strategies generates more accurate prediction estimates: if the unseen data is well represented in the training set, RCV may be more accurate. To address these issues, we introduced a simulated annealing approach, which is not limited by the characteristics of any particular clustering algorithm and allows us to construct test/training partitions with desired degrees of distinctness (a measure that quantifies how similar a test set and a training set are to each other). In the context of CV, the distinctness score can be thought of as an extra parameter which allows us to compare two prediction algorithms more accurately: instead of obtaining average accuracy performance of each algorithm and comparing these two numbers, one can compare their performance on test sets spanning a spectrum of distinctness scores, resembling unseen data of various degrees of similarity to the training set. As we showed in this study, two prediction methods (SVR and Elastic Net) had similar performance on less distinct partitions and a RCV evaluation would have concluded that these methods approximately perform equally well. On the other hand, more distinct partitions revealed that such a conclusion does not hold if the test sets are more distinct from the training sets and Elastic Net performs better.It is worth pointing out that the ability of a training dataset to provide accurate estimates of the prediction performance on unseen data depends on the heterogeneity of the training data as well: in a homogeneous dataset comprising of mostly similar conditions, any CV technique (even SACV) can produce partitions with only a limited range of distinctness scores. As a result, such a dataset may be most suited to generate context-specific predictions. However, even in such a case, utilizing SACV to generate partitions with a spectrum of distinctness scores may provide a better means to compare prediction performance of different algorithms.Another important point is that while the distinctness score is negatively correlated with the performance, the actual value of performance measure also depends on the size of the test sets. As a result, when comparing two prediction methods, one must ensure that the test set sizes are equal. This is yet another benefit of using SACV in comparing methods , since it can generate partitions with varying distinctness for any desired test set size, allowing to compare performance accuracies while fixing the test size and the distinctness score.19: glioblastoma multiformae (GBM), lymphoblastic acute myeloid leukemia (LAML), head and neck squamous carcinoma (HNSC), lung adenocarcinoma (LUAD), lung squamous carcinoma (LUSC), breast carcinoma (BRCA), kidney renal clear-cell carcinoma (KIRC), ovarian carcinoma (OV), bladder carcinoma (BLCA), colon adenocarcinoma (COAD), uterine cervical and endometrial carcinoma (UCEC) and rectal adenocarcinoma (READ). This dataset includes RNA-seq expression profiles of primary tumor samples in 3599 individuals and was downloaded from ‘https://xenabrowser.net/’. In addition, we obtained PanCan-normalized gene expression of four other cancers from TCGA: prostate adenocarcinoma , liver hepatocellular carcinoma , brain lower grade glioma , and cervical squamous cell carcinoma and endocervical adenocarcinoma . Moreover, we obtained a list of human TFs from the Animal Transcription Factor Database 20, of which 1239 were present in this TCGA dataset.We obtained mRNA expression data from The Cancer Genome Atlas (TCGA) corresponding to 12 cancers in the PanCan12 datasetWe used k-means clustering (k = 6) in order to construct clustered CV partitions. Clustering was performed on 864 samples included in our study. Each output cluster was then used as one CV fold. This procedure was repeated 10 times with different initial seeds for the k-means algorithm, resulting in 10 different CCV partition collections.Dmin and Dmax denote the minimum and maximum of these Euclidean distances. Then, given the Euclidean distance of two conditions D, their min-max normalized Euclidean distance is calculated as To calculate the min-max normalized Euclidean distance between two conditions (TF vectors), we first calculated the Euclidean distances between all pairs of conditions. Let TM using the expression of TFs in each condition as predictors of gene expression in that condition. The hyperparameters of LARS and Elastic Net were chosen using an inner 5-fold CV in the training set. Gene expression data used as the input to the regression algorithm was z-score normalized.LARS, Elastic Net, and support vector regression (SVR) with Gaussian kernel were performed in MATLAB22 was performed to generate CV partitions with varying degrees of distinctness; hence we used the defined distinctness score of a CV partition as the objective function. Starting with a random choice of conditions as the test fold and initial temperature equal to 1, in each iteration at a specific temperature, one condition from the test set was substituted with a condition in the training set. The distinctness of the new CV partition was evaluated and the substitution was accepted with a probability calculated based on the change in distinctness of the CV partition and the SA temperature of that step. After 500 iterations, the temperature was decreased by a factor of 0.98. This substitution procedure was repeated until the temperature reached the value of 1e-14, resulting in the 73423 accepted CV partitions. All of the accepted CV partitions from all iterations  were recorded. The first 50000 CV partitions produced by the SA algorithm were discarded and a total of 30 CV partitions with varying degrees of distinctness were uniformly sampled from the remaining SA-produced CV partitions.SAhttps://tcga.xenahubs.net/download/TCGA.PANCAN12.sampleMap/PanCan12.3602-corrected-v3_syn1715755.gz]. The source code files for distinctness calculations and simulated annealing-based partition collection generation are available at https://github.com/ShayanBordbar/Distinctness.The datasets analyzed during the current study are available in the UCSC Xena repository, [Supplementary Material


The aberrant alterations of biological functions are well known in tumorigenesis and cancer development. Hence, with advances in high-throughput sequencing technologies, capturing and quantifying the functional alterations in cancers based on expression profiles to explore cancer malignant process is highlighted as one of the important topics among cancer researches. In this article, we propose an algorithm for quantifying biological processes by using gene expression profiles over a sample population, which involves the idea of constructing principal curves to condense information of each biological process by a novel scoring scheme on an individualized manner. After applying our method on several large-scale breast cancer datasets in survival analysis, a subset of these biological processes extracted from corresponding survival model is then found to have significant associations with clinical outcomes. Further analyses of these biological processes enable the study of the interplays between biological processes and cancer phenotypes of interest, provide us valuable insights into cancer biology in biological process level and guide the precision treatment for cancer patients. And notably, prognosis predictions based on our method are consistently superior to the existing state of art methods with the same intention. Advances in modern technology have enabled us to measure the mRNA abundance at a whole genome scale, and also these have led to comprehensive catalogs of diverse molecular functions underlying biological processes in cancers. Therefore, it is essential to design a method with capability of quantifying biological processes by integrating gene expression profiles to reveal the general aberrations residing in cancers, as well as enabling the patient specific deregulations of biological processes.Many biological processes are rewired or reprogrammed during cancer initiation and progression. Therefore, identifying the involved biological processes and quantifying their levels of deregulation over a sample population are significant toward understanding cancer behaviors. Moreover, linking deregulation of biological processes in cancer patients with sensitivity to therapeutics that target the key components of these biological processes provides us an opportunity to guide personalized medicine5, but actually they do not provide any useful information on deregulation of that biological process in cancer. In order to guide personalized understanding of cancer biology, our method focuses on deducing independently, for every concerned biological process, a score that can represent the relative extent to which the pathway is deregulated in every individual sample.Nevertheless, there are at least two challenges. Firstly, in the era of high-throughput technology, we are constantly facing large-scale datasets enriched with information, but complex in the feature size and the intrinsic structure. And traditional methods using gene expression data in biological process analysis only target the single gene, such as oncogene or tumor-suppressor, while they cannot accurately and comprehensively unveil the variations as a result of combined effects of associated genes involved in a biological process. It is thus natural to transform gene level information into biological process level information so that we not only reduce the data dimensionality but also summarize the patterns of biological processes exhibited by the intrinsic structure of gene expression data themselves. Secondly, almost all the existing methods try to measure the activity of a biological process for entire sample6, as well as Pathway Recognition Algorithm using Data Integration on Genomic Models (PARADIGM)7. But the truth is that, at least for now, for some complicated biological process, we only have a rough understanding about its mechanism and relationship among its components, that is, we do not have a clear and complete map of the network structure. Moreover, lack of essential relevant data, such as protein data which PARADIGM requires, further limits usage of these methods in many occasions. Another framework does not rely on network information of biological process, such as Gene Set Variation Analysis (GSVA)8, which estimates variation of gene set belonging to a biological process over the samples. It starts evaluating differential expressed gene in each sample in the context of the current sample distribution, that is, constructing statistics of expression level and ranking these statistics for each sample on each gene, then condensing the statistics of gene expression level into biological process level scores by calculating sample-wise enrichment scores. However, GSVA is biased in scoring biological process where all genes have the same pattern, and yields less informative score for each sample9.For quantifying biological processes, there exist two popular frameworks. One of them characterizes the deregulation of biological process by deducing score for each sample based on network structure, such as PathOlogist10, to quantify the variations of each biological process for each individual sample. Our algorithm successfully bypasses the shortages of the conventional approaches, such as explicitly encoding phenotypes within biological process scoring algorithms, or restricting to detailed knowledge of the network, and further concentrates on summarizing patterns in context-specific manner. Principal curve generalizes the principal component line, aiming at passing the middle of the data cloud, and further provides a smooth one-dimensional curved approximation to a set of data points in Rp. The existing principal curve algorithms mainly take two following strategies: “top-down” strategy and “bottom up” strategy. The “top-down” strategy is usually used in traditional principal curve algorithm, it starts with a straight line, which is usually the first component of the whole data set, and tries to find the principal curve iteratively by minimizing the average squared distance between data points and corresponding projections to the curve. And the “bottom up” based strategy is applied in our LPC algorithm, it concatenates piece-wise curves that are determined by local neighborhood of currently considered data point. Comparing the two strategies, we find the second strategy seems to be more appropriate in finding principal curve, since the stretch of principal curve depends on the local distribution of data points rather than an initial line can effectively characterize complex data, such as circled and spiral data, even data with multiple branches. Our quantifying algorithm adopts the second strategy to construct principal curve characterizing the intrinsic structure of a biological process, through which we can obtain Biological Process Score (BPS) for each biological process and sample.In this article, we have developed an unsupervised learning algorithm called Local Principal Curve (LPC), which is based on the theory of Principal Curve (PC)To demonstrate the biological process deregulation scores obtained using our method indeed capture biologically and clinically relevant information in a sensible manner, we applied these biological process scores to model survivals of breast cancer as a case study. With the BPSs yielded by our method as inputs, we constructed a biological process based model that selected several biological processes are highly correlated with clinical outcomes, and these BPSs perform best in prognosis prediction in comparison to existing methods. Moreover, we found that the biological relevancies with compelling evidences between selected or cancer hallmark related biological processes and breast cancer have been referred to in many existing researches.The outline of proposed algorithm is shown in Fig. 11 and pathway-based deregulation score (PDS) matrix yielded by Pathifier12 respectively. Here, PAM50 is a gene set designed for subtyping breast cancer, and also has satisfactory performance in breast cancer prognosis13, and Pathifier is derived with the same intention as ours, but using principal curve method with classical “top-down” strategy, and was believed to be the best currently available method for deducing gene set level scores14.In this article, we first applied our method on a breast cancer dataset GSE3494 with abundant clinical information that includes 236 samples over 1330 biological processes, and obtained corresponding BPS matrices (see Method). To validate the effectiveness and robustness of our BPS matrix, we compared our BPS matrix with PAM50 gene expression matrix15 on the BPS matrix, PAM50 gene expression matrix and PDS matrix respectively with sample survival status as the response variable, and scores of each biological process or expression values of each gene as predictor variables. And then we calculated the significances of associations between the three matrices and various clinical outcomes. The scores for the three matrices and their associations with various clinical outcomes are shown as heatmaps in Fig. We selected those biological processes whose scores are significantly associated with breast cancer survival. To do so, we applied L1-LASSO penalized multivariate Cox proportional hazards (Cox-PH) modelCompare to PAM50 gene wise expression matrix and PDS matrix by the same survival analysis model respectively, BPS matrix is more prognostic for survival status, meanwhile, has high correlation with clinical outcomes. This conclusion can be confirmed by following two observations: 1) Dichotomized samples of high risk and low risk group through hierarchical clustering of BPS matrix have higher correlation to survival status , in comparison with PAM50 gene expression matrix  and PDS matrix . 2) Clinical factors such as ER status, PR status, p53 mutation status, lymph node status and grade, have stronger association with those two groups dichotomized by clustering BPS matrix , in comparison to PDS matrix  and PAM50 matrix .To further validate the effectiveness and robustness of BPS matrix, we introduce Prognostic Index (PI) value, which is logarithm of hazard ratio obtained from L1-LASSO COX-PH model, as a measure of survival risk for each sample (see Methods). Sample with higher PI value tends to have a severe survival status. Therefore, we evaluated the effectiveness and robustness of BPS matrix via the following two approaches involving in PI: 1) Estimating survival difference between low risk and high risk groups dichotomized by PI threshold, via calculating Wilcoxon log rank test p-value of Kaplan-Meier survival curves based on these two groups. 2) Calculating area under the curve (AUC) of ROC based on binary classification through treating PI as a likelihood of survival status.We applied leave one out cross validation (LOOCV) to assess the effectiveness and robustness of BPS matrix. As to the first approach, we trained L1-LASSO COX-PH model using training samples at each fold, and then predicted PI value for test sample using the coefficients fitted in the training model. Test sample was further labeled with either low or high risk through comparing its predicted PI to PI threshold, which was determined from the training model to match the ratio of alive versus dead samples of training and used to dichotomize the training samples into higher risk and lower risk groups. Then, we calculated Wilcoxon log rank test p-values of Kaplan-Meier survival curves based on these two risk groups for training result (2.95e-10) as well as LOOCV result (8.79e-5) respectively. By contrast, we obtained the Wilcoxon log rank test p-values from PAM50 gene expression matrix for training result 2.22e-3) as well as LOOCV result (3.76e-1), and from PDS matrix for training result (4.91e-9) as well as LOOCV result (9.1e-3) using the same procedure described above respectively. As to the second approach, we regarded PI as likelihood to survival status, and calculated AUCs for training result (0.78) as well as LOOCV result (0.69). In contrast, we also calculated AUCs for training result (0.71) as well as LOOCV result (0.47) based on PAM50 gene expression matrix, and for training result (0.74) as well as LOOCV result (0.62) based on PDS matrix, see Fig.  as well Moreover, we further evaluated the effectiveness and robustness of BPS matrix considering batch effect across datasets. We used one dataset (GSE3494) as training data, and two other independent datasets (GSE1456 and GSE4922) as testing data. As to the first approach, we trained L1-LASSO COX-PH model using the training data, and then predicted PI value for test data using the coefficients fitted in training model. All samples in the training data were divided by PI threshold into low risk and high risk groups, where the ratio of these two risk groups matches the ratio of alive versus dead samples in training data. And the same PI threshold was applied to dichotomize the testing data. We repeated the experiments based on the above two approaches and examined the results from two independent datasets, then found that the Wilcoxon log rank test p-values of Kaplan-Meier survival curves from BPS matrices of testing data (4.42e-2 and 4.79e-2) are more significant in comparison to those from PAM50 gene expression matrices (1.5e-1 and 3.26e-1) and PDS matrices (5.3e-1 and 7.65e-1), and the AUCs from BPS matrices of testing data (0.54 and 0.57) are better than those from PAM50 gene expression matrices (0.42 and 0.53) and PDS matrices (0.40 and 0.55). The BPS matrices still consistently show the best results both in significance of correlation with clinical factors and performance of prognosis prediction  subfamily of receptor tyrosine kinases. Supporting role in promoting growth of breast cancer cells, signaling by ERBB4 also has been suggested to induce differentiation and apoptosis of breast cancer cells28. The second one is REACTOME TRAF6 MEDIATED NFKB ACTIVATION, where NF-kB is a family of transcription factors that play critical roles in cell survival, proliferation, inflammation and immunity29. And, there are many evidences show the activation of NF-kB contributes to the breast cancer development and progression30.Then, we focused on biological processes about genetic information processing. The first one is PID CMYB PATHWAY, where c-MYB is the prototype member of a small family of transcription factors involved in cell survival, proliferation, differentiation and transformation, and it has been reported recently that c-MYB enhances the migration and invasion in breast cancer32. It has been demonstrated that the blockade of CTLA-4 augments endogenous responses to tumor cells, thus leading to tumor cell death in breast cancer33. And another one needs to be considered is REACTOME REGUALTION OF INSULIN SECRETION BY ACETYLCHOLINE, insulin of high level induces proliferative tissue abnormalities because of its strong anabolic effect of insulin, which results in stimulated DNA synthesis and cell proliferation34.And the rest of biological processes still remain strong associations to breast cancer. One of them should be noticed is BIOCARTA CTLA4 PATHWAY, where Cytotoxic T lymphocyte antigen-4 (CTLA-4) is a key negative regulator of T cell activation. Within hostile local microenvironment and in distant organs, immune cells such as T cell and their mediators are known to facilitate metastasis formationIn this article, we also applied our method on the same breast cancer dataset over 229 cancer hallmark related biological processes (see Methods), and then obtained corresponding BPS matrix. And we then did the same experiment on this BPS matrix of cancer hallmark related biological processes, even though the correlation with clinical outcomes and prognosis prediction are slightly inferior to those derived from BPS matrix of biological processes from MsigDB above, it still performs better in comparison to either PAM50 gene expression matrix or PDS matrix of cancer hallmark related biological processes  confers the trait of advanced or late stage cancer, which is induced by the redirection of TGF-β signaling1. The EMT program in breast cancer is also reported to be highly associated with cancer invasion and metastasis36, hence it is valuable to evaluate the related biological processes in our analysis. We then analyzed the N-acetly gulcosaminyl transferase pathway, whose synergistic effect with TGF-β to induce EMT in breast cancer is reported37. Moreover, SMAD and its downstream target signaling pathway is also highlighted, where SMAD acts as the intracellular effector of TGF-β signaling pathway38. In addition, the extracellular matrix cell to cell adhesion pathway (EGS_Extracellular_Matrix_Cell_Adhesion) and the global intercellular signaling pathway  are included, both of which are reported to be highly associated with the invasion and metastasis of breast cancer39. At last, we also checked the immune detection pathways, especially the humoral immune responses, since this biological process is widely reported to have the anti-tumor role and its deficiency is observed during the breast cancer progression40.In the following part, we checked 6 biological processes from above survival model that are highly correlated to the malignancy  of breast cancer. Firstly, the genome instability is one of the most fundamental hallmarks of cancer, while the base excision repair pathway is the “gatekeeper” biological process to ensure the genome integrity. It is widely observed that the deficiency of this biological process associates with the malignancy of breast cancer patientsIn this article, we propose a novel unsupervised algorithm based on local principal curve in acquiring intrinsic structure of biological process and quantifying corresponding biological process for each sample, which successfully bypasses the conventional method of explicitly modeling phenotypes of samples and network information of biological process within algorithm itself. The obtained biological process scores provide superior performance in survival analysis compared to scores yielded by Pathifier and expression values of PAM50 genes. Moreover, the selected biological processes in survival analysis are highly associated with breast cancer and even cancer hallmarks, and further can be exploited to target prognosis of breast cancer and even gain an insight into progress and development of breast cancer.Notably, our algorithm is highly potential in detecting the complicated structure of biological process, including spiral, bifurcation and even multiple disconnected branches. But in this article, we are constrained by gene expression data with insufficient samples, that is, there exist large gaps among different types of samples in some mapping spaces of biological processes. We will try to collect more samples in further study to more accurately and completely capture the subtle changes. And even if samples are adequate, they may be unevenly distributed in mapping space of a specific biological process, we cannot reach a comprise between characterizing subtle information and picturing the whole underlying structure of a biological process when using the same step and bandwidth in each movement of localized center of mass. Therefore, we will further work on improving our algorithm in selecting a data-adaptive parameter dynamically. As for values of the deregulation scores produced by our method, they can only be acted as an estimation of the relative extent to which the behavior of biological process in a sample deviates from the designated sample, we still cannot use these scores to mine the real changes, such as up- and down- regulation, or fluctuations over time in biological process. We will further make efforts to find a reasonable way to address this issue.43. Next, we will try to link our scores of the cancer hallmark related biological processes to their networks for each individual patient, and generally infer the process of the changes over variation of scores in these cancer hallmark related network, which will have substantial impact on understanding the mechanisms of tumorigenesis and cancer development, even guiding personalized treatment and prevention of cancer.Furthermore, there still exists some information cannot be reflected in individual omics data, such as gene expression data that we used in this article. It is necessary to consider extending our algorithm to integrate multiple omics data in a reasonable manner. More efforts have been made in modeling cancer hallmarks with their network for gain some insights into cancer biology, and they have generated many interesting results44, GSE145645, GSE492246 from public database Gene Expression Omnibus (GEO)47. All datasets are measured on microarray platform of Affymetrix HG-U133A, including 236, 159, 249 samples with survival information respectively, and GSE3494 has more abundant clinical outcomes, such as status of Estrogen Receptor (ER), Progesterone Receptor (PR), Lymph node, P53 mutation and Grade etc. The annotated gene sets are obtained from Molecular Signatures Database (MSigDB)48. We used curated gene sets collection (C2 collection), which includes 217 biological processes from BIOCARTA49, 186 biological processes from KEGG50, 674 biological processes from REACTOME51 and 253 biological processes from Pathway Interaction Database52. And all cancer hallmark related gene sets we used in this article are gathered from literatures and databases by ourselves (see Supplementary Data files).In this study, we collected three breast cancer datasets GSE349453 before our normalization procedure. And we reduce the batch effect across the datasets using Combat54. Each of normalized gene expression values is regarded as a degree to which a tumor sample deviates away from its corresponding normal samples in current gene expression scale. Besides, to avoid genes whose variations are attributable to noise, we exclude the genes whose variances lower than 25% of all genes.Firstly, the variation of expression values may be largely affected by gene wide range scales. Hence, genes of absolute smaller variations caused by its intrinsic lower expression scales, are usually overwhelmed by those genes of higher expression scales. To eliminate this bias induced by different scales of gene expression values, we normalize gene expression values of tumor samples by first subtracting each gene’s expression value of each tumor sample by the mean expression value of the normal samples for the corresponding gene, and then dividing each gene’s expression value of each tumor sample by the variance of the normal samples for the corresponding gene when normal samples exist, otherwise, we normalize gene expression values of tumor samples by calculating z-scores. Here, to reduce technological noise or other biases, the gene expression data are processed using limmaSp denoted by all genes involved in that biological process, but in the subspace S′p of Sp spanned by first k principal components from gene expression data of that biological process. Here, k is identified by the number of principal components along which the variances exceed by more than 10% of those in normal samples.In addition, it is noteworthy that some genes in the same biological process may be highly correlated, resulting in the redundant information that causes higher computation expense. However, some important information regarding to the functionalities of the biological processes may hidden in these genes. To counter this issue, we decide not to search principal curve for each biological process in its corresponding original space X1, …, Xn, with Xi ∈ Rp. Local principal curves (LPCs)38 are based on the idea that, at each point x ∈ Rp along a principal curve, the localized first principal component line forms approximately a tangent to the curve. Beginning with a starting point, x = x0 ∈ Rp, LPCs go successively through the data cloud, alternating between the following three steps:Calculating a localized center of mass γx of local weighted covariance matrix of x, j-th component of μx;Computing the first local eigenvector t0 to iterate from μx to x = μx + t0γx along the direction of γx.Using a predetermined step size μx falls below a preliminary threshold.Stop forwarding when the boundary of data cloud is reached, or the difference of the previous and the current center of mass Assume that we have a data set μx forms the local principal curve. Here, K and a positive definite bandwidth matrix et al.55. In this article, we arbitrarily choose 10% of range in each dimension for bandwidth matrix. Extensions to disconnected and branched curves are also considered in Einbeck et al.56, and easily implemented by using suitable multiple starting points. The issue of crossing and oversteering can be handled by using angle penalization57. As only points in the local neighborhood are considered in each iteration, the algorithm is quite flexible, and robust to noisy data patterns.The sequence of local centers of mass L local centers of mass Ig ∈ R denotes the domain of g, and g is a vector function with p coordinates, each smooth function shares the same parameter t. The parameter t corresponds to the projection index, whose variation traces all points along the curve. One ending point of the curve is chosen to be the origin corresponding to t = 0.t, by adding up the Euclidean distances between subsequent Computing a discrete, preliminary parameterization j = 1, …, p, interpolating the points For each dimension s within the support of the spline function, recalculating the parameter using the arc length,For each value of For a fitted LPC consisting of g(t) = μ(s). It should be noted that smoothing is not involved in previous steps. Once this parameterization is established, each data point xi, i = , …, n can be projected onto the curve through finding the point on the curve which is nearest to it in Euclidean distance, meanwhile, yielding the projection index ti.and setting p-dimensional space Now, we suppose that there exists a x0 . Once such a principal curve is found, we project each data point xi representing i-th sample onto fi on the curve, and then calculate projection index ti along the curve between fi and the projection of one end point of curve f0 . Then, we normalize all projection indices as biological process scores BSi, each of which measures the degree to which a given biological process of sample i is altered by tumor  is a hazard function only depends on time as baseline, β is a coefficient vector of corresponding features. Subsequently, the relative hazard ratio between any two features Xi, Xj  is independent on time and only determined by the differences of two features:Xk (1 ≤ k ≤ n) at one time, PI is relative constant to other samples, and can be represented as following:In this article, we use the COX-PH model for survival analysis with the assumption that log hazard ratios are constant over time for features. Suppose that we have a matrix Supplementary InformationDataset 1


Quantitative molecular biology remains a challenge for researchers due to inconsistent approaches for control of errors in the final results. Due to several factors that can influence the final result, quantitative analysis and interpretation of qPCR data are still not trivial. Together with the development of high-throughput qPCR platforms, there is a need for a tool allowing for robust, reliable and fast nucleic acid quantification.http://quantgenius.nib.si), an open-access web application for a reliable qPCR-based quantification of nucleic acids. The quantGenius workflow interactively guides the user through data import, quality control (QC) and calculation steps. The input is machine- and chemistry–independent. Quantification is performed using the standard curve approach, with normalization to one or several reference genes. The special feature of the application is the implementation of user-guided QC-based decision support system, based on qPCR standards, that takes into account pipetting errors, assay amplification efficiencies, limits of detection and quantification of the assays as well as the control of PCR inhibition in individual samples. The intermediate calculations and final results are exportable in a data matrix suitable for further statistical analysis or visualization. We additionally compare the most important features of quantGenius with similar advanced software tools and illustrate the importance of proper QC system in the analysis of qPCR data in two use cases.We have developed “quantGenius”  contains supplementary material, which is available to authorized users. The immense potential of quantitative molecular biology in life sciences is challenged by inconsistent approaches for control of errors in the final results. Due to its performance characteristics and general applicability, quantitative PCR (qPCR) has become the golden standard method for the quantification of nucleic acids. Although with the help of laboratory automation, qPCR data generation has become easy and fast, quantitative data analysis and interpretation is still not trivial due to several factors that can influence the final result. To ensure high quality of results and allow for potential reproduction of experiment, the Minimum Information for Publication of Quantitative Real-Time PCR Experiments (MIQE) guidelines have been proposed .q values, which are further used for the quantification of the nucleic acids. It can be performed by either a standard curve or a comparative approach . Both approaches are relative, but each is based on its own assumptions [q and the logarithm of the initial amount of template DNA. Test sample copy numbers are calculated from the linear regression of the standard curve, assuming equal amplification efficiencies for the standard and test samples [q) relies on a direct comparison of the Cq values and assumes equal and 100% efficiencies of the target and the reference gene. However, the amplification efficiencies between different genes analysed can differ which makes the ΔΔCq method unsuitable in many cases [qPCR is used to measure the quantity of target DNAs in a given sample through repeated cycles of DNA amplification. The cycle at which the observed amplification-derived fluorescence first exceeds a certain threshold is called the quantification cycle (Cq). The analysis starts with the examination of the amplification curves and initial assessment of their quality, followed by the determination of the Cumptions . In the  samples . When re samples  whereas  samples . The secny cases . Consequny cases , they caThe efficiency of PCR amplification is considered as one of the most important parameters in qPCR analysis, as it strongly influences the final result , 6. The Normalization controls for variations in extraction yield, reverse-transcription yield and efficiencies of amplification, thus enabling comparisons of nucleic acid concentrations across different samples. Various normalization strategies and reference genes selection algorithms have been proposed with the common guideline that several validated reference genes should be used for normalization (reviewed in ).http://quantgenius.nib.si), the only qPCR data analysis solution that integrates a QC-based decision support system (DSS). Among other features, it includes a control of inhibition in individual samples which is extremely useful when working with difficult samples, such as environmental or plant samples. In this way, it helps the scientist to obtain reliable results in a fast and high-throughput manner and thus provides the basis for biologically meaningful data interpretation.Although numerous commercial and open-access software tools for the analysis of qPCR data exist  data import, 2) interactive calculation of target and reference genes copy numbers and normalization to reference genes with implemented QC-DSS and 3) export of final results in a gene-sample matrix format  are imported by pasting pre-formatted tab-delimited data into the input form. In this way, sample data analysed with one or two sample dilutions and any number of technical replicates can be processed. Data for the standard curve, which can be either actual or relative copy numbers, are imported separately  can be converted to a format suitable for the import using the “Fluidigm data prep tool”, available on the quantGenius website.All imported data are automatically checked for consistency  to prevent wrong calculations due to incorrect imports (for example copy-pasting errors).In quantGenius, a standard curve quantification approach is implemented, which allows for the calculation of comparable copy numbers on multi-plate experiments, when the same standard curve is used on all plates. For optimal transparency of the process, the calculations are performed in several steps Fig. , differiTarget gene copy numbers are normalized to reference gene copy numbers, or in the case of several reference genes, to their average  or two-dilution calculation approach is chosen Fig.  and hierExtraction control. For each reference gene assay, CqExtC, a Cq value indicating a valid nucleic acid (DNA or RNA) extraction procedure is defined by the user ensuring that only good quality data is used for calculations. By default, the CqExtC is set to 34, therefore rarely affecting the quantification. Based on the assumption that the reference genes are highly expressed, the users can, however, lower this threshold to identify outlier samples. If all the reference genes fail this criterion, the target gene final result is not calculated . For each target gene assay, the Cq at the LOQ (CqLOQ), specifying the lower limit of the quantification of the assay is defined by the user, either based on previous in-house validation data (if available) or estimated from the experiment’s standard curve. On the other hand, the LOQ can be recognized by quantGenius as high variability (CV > 30) between the replicates’ copy numbers, arising from pipetting stochasticity, assuming that the true pipetting errors have previously been manually removed. In the simple calculation, the final result for samples below LOQ is imputed based on the CqLOQ and all sample reference gene data . If all reactions of the sample for a target gene are missing Cq values, indicating that the target DNA levels in the sample are under the LOD of the assay, then the final result is imputed based on the CqLOQ copy numbers of all sample reference gene data  format  quantification analysis.q values) and basic experimental details are available in Additional file quantGenius two-dilution quantification with the default QC settingsstandard curve quantification approach without any QC-DSSq approach [commonly used ΔΔCapproach , using oFor the gene expression use case, a subset of qPCR data from our previously published experiment , analysir > 0.99) for both target genes. Nevertheless, the power of quantGenius is shown in the case of individual samples with low gene expression values and sub-optimal amplification efficiencies.The relative copy numbers obtained in the three approaches are presented in Fig. q values near CqLOQ and high inter-replicate CVs), which resulted in high copy number variation (CV > 50) between different quantification approaches  in the samples is quantified as a ratio of the transgene and reference gene (soybean lectin) copy numbers, both calculated from the standard curve of the reference material with known GMO content . In the The paper presents a web application for quantification of nucleic acids, integrating unique QC-based DSS Fig. , built bq results was really high . Moreover, low amounts of DNA in the qPCR reaction can increase the measurement uncertainty due to the high variability of quantity estimations caused by the occurrence of stochastic effects, therefore only the reactions above the LOQ can be accurately quantified . The real models . In the l models . Therefoq values from technical replicates can be an indication of an unstable assay, a pipetting error or stochastic effects due to the low amount of DNA in the reaction [Low reproducibility of the Creaction . The latreaction  which alq values of two dilutions of the same sample to identify outlier samples with suboptimal efficiencies. As quantification is in those cases not accurate, no result is given for those samples. The presented approach is associated with slightly higher cost of wet-lab analysis , but on the other hand it greatly increases the quality and reliability of the data, especially in samples where the presence of inhibitors is expected, such as plant samples, food and feed samples, environmental samples, microorganisms grown in complex media etc. [There is currently no consensus on how sample specific PCR efficiencies should be calculated and used for robust quantification. Although the individual sample amplification efficiencies determined from the amplification curves increase the random error of qPCR quantifications , the inddia etc. , 11. Thidia etc. , 4. Howehttps://bio.tools) [To promote quantGenius use within the scientific community, the application was is registered in the ELIXIR Tools and Data Services Registry (o.tools) . Future o.tools) .As opposed to black box solutions, quantGenius was designed by biologists with ease of use, flexibility and transparency in mind. It is an intuitive and easy to use tool for qPCR data organization, analysis and decision support in various qPCR applications. The integration of QC-based DSS makes it unique and enables researchers to spend more time for interpreting the biology behind the results than analysing the data.Additional file 1:Schema (ER diagram) of the database and data organization in quantGenius. (PDF 166 kb)Additional file 2:Equations used in quantGenius workflow. (PDF 122 kb)Additional file 3:qPCR data import screenshot. (PDF 53 kb)Additional file 4:Example of the export of results and intermediate values for an individual gene. (XLS 41 kb)Additional file 5:quantGenius performance validation: calculations in Excel. (XLS 50 kb)Additional file 6Experimental data of the gene expression use case. (XLS 71 kb)Additional file 7:Comparison of gene expression values calculated by quantGenius and other standard methods. (XLS 49 kb)Additional file 8:GMO quantification use case. (PDF 228 kb)


We studied the transcriptome landscape of skin cutaneous melanoma (SKCM) using 103 primary tumor samples from TCGA, and measured the expression levels of both protein coding genes and non-coding RNAs (ncRNAs). In particular, we emphasized pseudogenes potentially relevant to this cancer. While cataloguing the profiles based on the known biotypes, all the employed RNA-Seq methods generated just a small consensus of significant biotypes. We thus designed an approach to reconcile the profiles from all methods following a simple strategy: we selected genes that were confirmed as differentially expressed by the ensemble predictions obtained in a regression model. The main advantages of this approach are: 1) Selection of a high-confidence gene set identifying relevant pathways; 2) Use of a regression model whose covariates embed all method-driven outcomes to predict an averaged profile; 3) Method-specific assessment of prediction power and significance. Furthermore, the approach can be generalized to any biological system for which noisy RNA-Seq profiles are computed. As our analyses concerned bio-annotations of both high-quality protein coding genes and ncRNAs, we considered the associations between pseudogenes and parental genes (targets). Among the candidate targets that were validated, we identified PINK1, which is studied in patients with Parkinson and cancer . In 10% of melanoma cases, there is a positive family history, i.e. a risk factor significantly contributing to disease occurrence2. At a genetic level, multiple genes are naturally expected to interactively exert direct and/or indirect influences on many biological processes and pathways, suggesting possible signatures of the disease. One of the first discovery efforts in melanoma transcriptome studies revealed the emergence of genomic rearrangements behind novel gene fusions and identified somatic mutations responsible for disease progression3. Further RNA-Seq analyses have provided in-depth screening for a variety of transcript structures, by studying human samples4 or cell lines5. RNA-Seq is the standard technique that is used to characterize cancer transcriptomes while measuring known transcripts and predicting novel transcripts11. Additionally, RNA-Seq expands the detection power of transcriptome profiles by identifying and elucidating a variety of non-coding RNAs (ncRNAs)13. Following the seminal findings of two large-scale projects, ENCODE14 (https://www.encodeproject.org/) and GENCODE15 (http://www.gencodegenes.org/), the general attention has been gradually diverted away from genes and concentrated over other biotypes, such as long intergenic ncRNAs (lincRNAs), antisense RNAs, pseudogenes etc. In most cases the identification of potentially transcribed bio-entitities was inferred from locus-specific (EST) and wide-spectrum experiments, then referring transcriptional activity to tissue specificity and establishing association with active chromatin state and promoter regions.Melanoma, the deadliest of skin cancers, is a malignant and highly metastatic tumor with growing incidence of about 20 per 100,000 individuals in Western countrieshttp://cancergenome.nih.gov/) and its entire collection of 103 primary SKCM samples. As TCGA contained only one control sample for SKCM, we retrieved other reference melanocyte samples in the GEO repository (http://www.ncbi.nlm.nih.gov/geo/). In order to identify both gene and ncRNA signatures, we built an ad hoc methodological pipeline to deal with the observed biotype variety and expression heterogeneity. Pervasive transcription has been widely documented, involving multi-exonic transcripts that map to intergenic regions with specific interactions and functions22. The ability to detect ncRNAs suffers from low read density coverage, thereby resulting in non-uniform coverage of transcript abundance. This latter is generally >100-fold higher in exons than in introns or intergenic regions. However, despite the limitation of more/less abundant transcripts represented by many/few reads, respectively, lowly expressed transcripts can be widely detected by RNA-Seq across many cancers.We performed a systematic exploration of the skin cutaneous melanoma (SKCM) transcriptome landscape by accessing the TCGA repository  showed the signature of 2082 distinct pseudogenes4. Several aspects of pseudogenes involve uncertainty with regard to identification, function, annotation, role in cancer26. We believe that knowledge gains can be derived from the use of computational approaches29. For example, the associations between pseudogenes and so-called parental genes, i.e. the functional genes with the highest sequence similarity, can be usefully investigated. One reason is that the complexity of such relationships goes far beyond the possible presence of correlation between expression levels. The latter is quite controversial because inconsistently supported by evidence. Based on the analyzed samples, we have provided an empirical assessment of such patterns both at systemic scale and at local level of specific associations31. Our integrative computational approach involves three main steps, which we applied sequentially to the SKCM transcriptome data.i)RNA-Seq analysis, performed by multiple methods. The results were then integrated to achieve a large consensus of differentially expressed (DE) bioentities, either differentially expressed genes (DEGs) or DE ncRNAs;ii)Ensemble statistical modeling. This step aimed to recover salient features that remain hidden in the consensus data. It reconcileed the profiles obtained from the different methods into a high-confidence inferable coreset of model-classified bioentities;iii)Bio-annotations (pathways and gene families), organized into curated lists of pathway and GO categories. These were supported by other classifications addressing the potential of genes to play a role in cancer .The pseudogene participation to the transcriptional activity deserves a special attention. Thousands of pseudogenes have been identified in the human genome, many with critical regulatory functionsWe performed two complementary methodological steps: a) The identification of pseudogenes and corresponding parental genes, followed by their profiling and by an assessment of their relationships; and b) The functional validation of selected DEG candidates to verify both reliability and effectiveness of our ensemble approach to generate testable hypotheses. The rationale for enabling an ensemble inference approach is that multiple RNA-Seq methods can be used to both quantify and profile the expression levels of transcriptome biotypes, but limitations are present with each method. Clearly enough, studies centered on specific methods are subject to risk of bias. Among the alternative strategies that can be used to overcome the limitations, some establish an efficient practice by combining results from more than one method. It is also possible that contradictory results may emerge from using methods operating different quantifications and normalizations. Our evidences demonstrate exactly this aspect. In particular, a method-driven selection of different and separated DE-space regions is obtained, and the significance is assigned to detections that are in large part not shared. To increase the consistency of the results, evidences should therefore be consolidated without aiming necessarily at a consensus. This goal can be achieved without over-penalizing disparate evidences through the ensemble modeling approach that we introduce here and demonstrate to be reasonably efficient and accurate.Figure At a biological level our primary interest is to detect the DE profiles of biotypes forming the SKCM transcriptome landscape Fig. , left. S32 aimed to yield transcript lists have not been used in this work, which is focused on gene-centric analysis. Thus, isoform detection and alternative splicing are not considered in this work.A summary of outcomes is presented in Fig. 33., NOISeq34., LIMMA35, CuffDiff36, and the licensed GeneSpring (V 13.0) . These criteria depend on specific factors, such as distributional hypotheses, built-in normalization procedures, parametric settings, multiple corrections, etc., all exerting influence over the final detections. Generally speaking, local rather than global data features can bring good approximation power for inherently heterogeneous data, but likely poor generalization power. Note also that complex experimental data may lack robustness against possible under-sampling effects, which means that the spectrum of data features may be inadequately represented. Lastly, among the factors that exert negative impacts on inference, there are noise, algorithmic errors, and uncertain annotation of the bioentities  because the ensemble LRM models neutralizes them by assigning non-significance to the corresponding regression coefficients. Finally, gene and ncRNA signatures derived from each of the DE profiles required some control of variability, and we applied thresholding through the boxplots and the IQR measures applied to the estimated LRM coefficients. The stringency of the assigned thresholds affects the size of our final selections.Following the ensemble modeling strategy, it is important to understand how the LRM regions affect the system’s predictive performance. In particular, what method should be used for response and what for predictive purposes in the regression setting? To answer this question, we chose to run regression model multiple times according to the rotation of variables serving each role, response vs regressors. This scheme is known as Alternating Regression ModelFigure 38). Formed from the DE parental genes associated with expressed pseudogenes (including DE), the network consists of interacting proteins retrieved from STRING V10.039 (www.string-db.org), also annotated with GeneCards: the Human Gene Database40 (www.genecards.org). Given an intermediate level of confidence (0.4), from both experimental and database knowledge, these interactions occur between proteins relevant for cell proliferation, adhesion, motility, morphogenesis, together with cell cycle and oxidative stress.Interesting evidences appear from the network in Fig. ATP1B1-ESAM-VWF-TF-PROS1-GJA1-TUBA4A-CDC20-PLK1-RRM2-NCAPG; PINK1-PRDX3-PGAM1-ENO3; CTU1-PPA1, HLA-B-GBP1; PLIN2-CDC42EP3-RHOQ-EXOC5-OPHN1-GNB4-HTR7. To some extent, these are all processes relevant to cancer, implying potential regulation underlying the target parental genes represented in the network and exerted by the either expressed or associated DE pseudogenes. When only DE pseudogenes are considered, we could observe a correlative scatter , being critically involved in many cellular processes, including energy production, metabolism, redox control, and programmed cell death. Their dysfunction is identifiable in numerous human diseases, including cancer46. Thus, PINK1 is essential for survival and a potential drug target due to its mediating role towards PTEN47. Despite the expression of PINK1 is up-regulated in melanoma with high metastatic potential, the molecular mechanisms by which PINK1 protects cells against apoptosis are unclear Notably, PINK1 is among the target parental genes that we have validated (see below).Interestingly, the PTEN- induced putative kinase 1 or PINK1, interacting with PRDX3, appears annotated with oxidative stress49. Two groups were obtained from both mutations, and are reported in Supplementary Tables 50. Mutations in NRAS, occurring at limited frequency compared to BRAF, were considered too. These two mutation types are almost always mutually exclusive. We accounted for the fact that approximately 50% of melanomas of all clinical types present an activating mutation in the BRAF oncogene, while about 20% of the cases show an activating mutation in NRAS, leading to the activation of the MAPK signaling pathway .Among the mutations involved in melanoma pathogenesis, those repeatedly identified from BRAF and NRAS oncogenes have been investigated3. The raw reads of SKCM samples were downloaded from TCGA site and were mapped on human genome using TopHat v2.0.11. SNPs were called by GATK 2.5-2-gf57256b. Non-synonymous mutations on V600 of BRAF were detected on 57 cases, but 19 were then discarded due to the low sequencing depth (<5x) on the mutation site, thus remaining with 38 cases. Non-synonymous mutations on Q61 of NRAS were detected on 17 cases, but 2 were then discarded due to the low sequencing depth (<5x) on the mutation site, thus remaining with 15 cases. Also, 33 cases have revealed no mutation on either V600 of BRAF or Q61 of NRAS, thus making the mutation-free group. The DEGs were detected by CuffDiff v2.2.1, with cutoff q-value < = 5% and log(FC) >1.2 between the BRAF-mutation group and the mutation-free group, and between the NRAS-mutation group and the mutation-free group. Overall, 1254 DEGs were detected in the former comparison, while 716 DEGs were detected in the latter one.Previously, whole-genome sequencing of 25 metastatic melanomas and matched germline DNA revealed 11 genes significantly mutated, for BRAF in 16 samples and for NRAS in 9 samplesWe considered annotations from GO  is therefore currently considered a primary therapeutic target. The PI3K (phosphinositide 3-kinase)–Akt pathway is studied to decipher molecular pathogenesis, heterogeneity and resistance mechanisms, offering emerging leading targets for compounds55. Melanomagenesis involves ECM signaling, p53, RAS/RAF/MEK/ERK, among others. It is also interesting the presence of the Complement system , an essential element of the innate immune response that reacts as a first line of defense against pathogens and in response to molecular patterns associated with abnormalities in host cells and extracellular environment56.By looking at the LRM terms Table , right, We expanded the annotations to include gene families and biotype decompositions  or the cell adhesion pathway (MPZ), for kinases  or for tumor suppressors (FH). The chosen candidates are nine protein coding genes, six of which are parental genes. Using rtPCR we confirmed down-regulation for SGK1, NQO1, PIK3CD, FH, TGFBR1, TWF1, PINK1 and up-regulation for MGP and MPZ. Note in particular the case of NQO1, initially not validated according to the consensus, but then validated according to LRM, this of course depending on the different selection operated in the two cases.We first considered consensus data Fig. , and mad57. Our contribution turns to the search for possible ways of selecting evidence that may yield a solid knowledge base specifically for melanoma, but not limited to this cancer.Knowledge advances in cancer genome and transcriptome studies depend, among other factors, on the integration between experimental and computational resources and skills. Such type of integration operates through data-customized bioinformatics pipelines. Despite a substantial variety of proposed methods, careful literature review suggests that a restricted number of them are routinely involved in major applications, thus making them standard approaches. Such process generates a sort of biased automatism through which the confidence assigned to results impacts cancer studies, but is not supported by comparative evaluations between methods or tests measuring sensitivity of results. Literature has devoted attention to different normalization techniques, for instance, without demonstrating solid criteria for preference between methods. The field keeps evolving, naturally enough, but with uncertainty about our detection accuracy. Additionally, and in cancer studies especially, some of the existing tools may be too conservative, and new supersets of data could be needed to cross-validate analyses and findings. This consideration is applicable to other studies58. The correlation patterns between intronic pseudogene expressions and unrelated host gene expression values were not revealed, and no correlative patterns with their parental genes appeared from the 10 melanoma samples that were included in the study58. Correlation between pseudogenes and parental genes was instead found for about 65% of pairs to be positive, depending also on the number of predicted miRNA targeting sites23. Considering another scale, it has been demonstrated a combined regulative action of pseudogenes over a miRNA target, the tumor suppressor gene PTEN expression, through the mechanism known as ceRNA (competing endogenous RNA)60. Overall, from both biological and computational viewpoints, the sequence similarity between pseudogenes and parental genes contributes to the complexity of both sequencing and gene expression profiling, and even validation. This global complexity calls for additional research61. We have indicated association patterns showing a certain consistency. However, limitations are naturally present and they mainly concern the fact that detecting pseudogene transcripts is a complicated matter with a few critical factors to keep in mind:Technological limitations: they affect the real detection power and may act as confounders for predictive algorithms, especially when the similarity of pseudogenes with parental genes is high;RNA-Seq presents read mapping uncertainty causing loss of both information and coverage;Computational problems with regard to alignment, i.e. reads that align to multiple genomic locations. Assembly limitations: a limited resolution and the fact that transcript variants from the same gene may have exons in common, can cause ambiguity difficult to resolve;RNA-Seq as only abundant transcripts are fully assembled;Sequencing depth: it is well-known that low or broadly expressed genes are not strongly supported by Other problems come from sequencing non-uniformity, presence of novel isoforms , quantification of expression levels, and harmonization of data normalized according to various criteria, for instance form experiments with different sequencing depth.Additionally, we focus on protein coding genes and ncRNAs, and in particular pseudogene-parental gene associations that were retrieved from the SKCM data. The relevance is clear: despite pseudogenes are only dysfunctional copies of protein-coding genes, pseudogene transcription has a pervasive presence in the human genome62. Ensemble modeling improves results by targeting a multitude of significant evidences, which here are the detections obtained from RNA-Seq data, and it achieves superior accuracy than any individual method. Benefits have been shown here in terms of bioannotations , and validations.Following the ideal path of transforming observations (data) into model outcomes (predictions) to reconcile system’s dynamics, a limitation is that moving from observations to inference, and then to validation, is not usually a linear path. In our view, the strategy of moving from a low- to a high-confidence set of DE bioentities develops along two routes: a) Empirical approximation through PCA, i.e. projecting the data to perform de-noising and extracting the prevailing signal in terms of variability; b) the path of LRM, which exploits the prediction power of each method according to its significant contribution to a response profile. Therefore, we may aim at approximating with acceptable accuracy the recoverable system’s dynamics63.). However, we have proven that the stability of the system can be promptly checked, and corrections can be made to improve the overall performance. Our regression framework simply stacks together all detections in a matrix with rows assigned to the distinct measurements (outcomes) and columns hosting the ‘explanatory variables’ (predictors). The contribution of each method is therefore mediated by the linear combination which determines the optimal prediction for each DE entry listed as the ‘dependent variable”. This way, an ensemble predictor built-in in a regression framework can be provided by averaging effects across the rows such that all model variables are re-modulated by significant coefficient estimates aimed at tuning the prediction power.Naturally enough, LRM encounters known limiting factors, such as the possible violation of linearity, an excess of variability across the covariates, and the risk of overfitting. Methodological solutions for such problems that go beyond naïve linear model approaches are well-known , but might involve more subtle aspects, such as redundancy or compensatory effects between sets of parameters65. It is also likely that both important and not important parameters may be often hard to separate.The analysis of ensembles of models which vary in parameter values is a powerful strategy for reducing ambiguity in systems biology and medicine. The model ensemble may encompass different structures to capture the systems dynamics. Whether these dynamics are stylized or simplified is relevant, as cancers are very partially known complex systems. Therefore, models may be sloppy66, and in order to solve complex problems such as dimensionality reduction and identification of coresets, being these gene sets for example, an effective strategy requires the concentration of the total information into salient data features. Although sub-optimal , this idea works under simple model assumptions, such as those here enabled by regressions. The great value of the performed analyses is to increase the ‘data strength’ and through this step obtain a high-confidence compendium of selected bio-entities.To conclude, we recall a very useful methodological principle that we have pursued through the suggested ensemble approach: along the path of ‘weakening inference algorithms’https://tcga-data.nci.nih.gov/tcga). 2 control samples were also downloaded from the Gene Expression Omnibus (GEO) repository67, from a study of with data accessible at NCBI GEO database with accession GSE3716968. The total number of 105 samples were downloaded in FASTQ format69.103 primary tumor Cutaneous Melanoma (SKCM) samples were downloaded from the Cancer Genome Atlas (TCGA) CGHub repository  using the TopHat 2 spliced aligner71. The human genome reference sequence was prepared for mapping by collating and indexing the 25 standard chromosomes  from Ensembl. The ‘bowtie2-build’ indexer program from Bowtie272 was used as the indexing step and the TopHat2 spliced mapper was used to independently map each of the sample reads from TCGA and GEO. Tophat2 requested a GTF annotation file (http://mblab.wustl.edu/GTF22.html) from Ensembl. After mapping, Samtools73 was used to process the output files from the 105 samples, thus converted to a format (.BAM) suitable for downstream quality assurance and analysis steps.All sample reads were mapped to the EnsemblEach sample was inspected to detect possible anomalies. Issues emerged with the percentage of mapped reads in two of the TCGA tumor samples, therefore excluded. The residual 101 TCGA tumor samples, and the two controls (from GEO), were thus analyzed.Mapped reads for each sample were sorted and binned based on their mapping quality (MAPQ) values. SamTools was used to filter reads based on MAPQ values. TopHat2 bins read alignments based on 5 MAPQ classes: values of 0, 1, 2, and 3 are assigned for read alignments with multiple hits of 10, 4-9, 3, and 2 locations, respectively; a value of 255 is for unique alignments. Only unique read alignments for each sample were extracted from the output files, and used for downstream analysis.74. We applied two constraints: genes showing an FPKM greater than 1 and showing >10 reads in at least 10% of samples were included for downstream analysis, in both conditions. Different combinations and permutations of FPKM and number of reads at variable percentages were tested  and intronic reads (reads mapping to both exons and introns). A table of read count values was created for downstream analysis, containing genes as rows and samples as columns  Reads mapped to the pseudogene sequence and not to the parental gene; 2) Reads mapped to both the pseudogene and to the parental gene, but with lower sequence similarity (<90%). Target parental genes were identified by aligning the pseudogene sequences using BLAST against a database of the protein coding CDNA sequences from Ensembl (v. 72). The best hit matches for a pseudogene sequence were selected based on e-value scores, and the best overall hit for a pseudogene was selected as its parental gene. BLAST results with e-value scores can be found in Supplementary Table 77, GeneSpring (http://www.genomics.agilent.com/), and CuffDiff. The chosen read count methods were DESeq and NOISeq (both in Bioconductor). Each method was run using the standard protocol and parameters available in their documentation, and output p-values were corrected for multiple comparisons using false discovery rate (FDR).FPKM and read count values were used for testing relative differential expression between the TCGA primary tumor samples and the GEO controls. Five (5) Methods were employed with different (FPKM or read counts) units as input. FPKM-based chosen methods were: limma from BioconductorDESeq. The R package (v. 1.18.0) performs analysis of read counts from HTSeq by estimating variance-mean dependencies, normalizing by an estimate of the effective library size, and testing for differential expression using a negative binomial distribution.NOISeq. The NOISeq package (v. 2.8.0) is a non-parametric method for analyzing read count data from HTSeq. There are no assumptions about data distributions, and normalization of count data is by using a trimmed mean of M-values (TMM)60.CuffDiff. The CuffDiff software (v. 2.2.1) models the variability of a gene’s RNA-Seq measurements by considering RNA-Seq fragment mappings and the gene’s splicing structure. CuffDiff normalizes the library size by using a quartile normalization, and estimates the replicate dispersion with a Poisson model.GeneSpring. The GeneSpring software (v. 13.0) (Agilent Technologies) analyzed FPKM values obtained from CummeRbund. A standard workflow for expression data was used, and no further normalization was applied. Tests for differential expression were calculated using a Moderated T-Test.limma. The limma package (v. 3.22.6) analyzed the FPKM data from CummeRbund. The “classic” workflow was performed, i.e., no ‘voom’ transformation was applied. Tests for differential expression were calculated using a Moderated T-Test.Each of the differential expression methods pooled the two conditions tumor and control together (Supplementary Tables 78 plug-in for Cytoscape79 (http://apps.cytoscape.org/apps/cluego) The ClueGO db for Gene Ontology (GO)80, Kyoto Encyclopedia of Genes and Genomes (KEGG)81, WikiPathways82, and Reactome83 were used for pathway enrichment analysis with default values, with the GO analysis being run independently from the other databases. Several molecule pathways are associated with melanoma, and some with clinical relevance .A merged matrix of log2(FC) from the differential expression results of all methods was created and used as input to LRM. The matrix consisted of genes (rows) and algorithms (columns), with each cell containing gene or ncRNA values. Values that were not identified by a method were flagged as missing in the matrix. The standard form “y ~ model” is applied, by the R function “lm” designed to fit the LRM according to:The above simply denotes that the outcome is assigned by one method, say CuffDiff, establishing the model response variable associated to explanatory variables forming linear combination of values obtained with DESeq, NOISeq, Limma, and GeneSpring (i.e. the other methods). The placement of such methods as response vs explanatory variable is subject to rotation, according to the idea of Alternating Regression Modeling. Predictions by the model were obtained using the R function “predict.lm”, then filtering by a suitable multiplier of the interquartile range (IQR). Preference went to predicted values outside 3*IQR range.n-by-p data matrix D, rows of D correspond to observations and columns correspond to variables. The coefficient matrix is p-by-p. Each column contains coefficients for one PC, columns in descending order of component variance. Equivalently, the PCs return information contained in the projected data through the correspondingly estimated coefficients, with the highest relevance assigned to the direction of maximal variance . In our setting, the tumor baseline signal is contrasted with the control data, and the noise is reduced by shrinkage operated from the boxplots. Therefore, each method is subjected to extraction of the first component (PC1), and a set of boxplot outliers is obtained in each case.In principle, PCA primarily finds the data directions explaining most of its variation. A dataset D, normalized into D’, has covariance matrix computed as COV(D’), and its eigenvectors  represent the PCs. The principal component coefficients, also known as loadings, are to be found. Given, say, the With regression, thresholding is applied via the interquartile range or IQR, a measure derived from boxplots. Optimality would imply some assumptions, a main one being predictors’ independence. This assumption does not usually hold, and sub-optimal conditions apply. As our setting presents intersections among the identifications obtained from five methods (e.g. overlapping sets), the emerging gene or ncRNA signatures must be confronted with the inevitable collinearity between regressors, i.e. the method-dependent profiles. Such redundancy may be reduced by enforcing shrinkage to the predicted values, through a robust selection rule, i.e. IQR × 1.5 (2 or 3). The selections represent linear combinations of multiple profiles.There might be a subset of predictors relatively more important than the rest. One strategy is to insert penalties in the model, useful to control the number of predictors involved. As an alternative, we proposed an ARM model type, one working by rotating the dependent and independent variables. Once an initial response is identified with one of the variables, the residual variables play the role of regressors/predictors for the response. It is a sort of cross- or hold-out validation principle applied to assess the goodness of fit of each method. Repeating the regression generates vectors of fitted values that need to be averaged at the end, thus yielding the final consensus DE vector. Note two aspects: (a) The dimension of each response varies depending on the profile to predict. The matrix of regressors will adapt to the response’s dimension by only matching selected response values; (b) Estimates of the regression coefficients guide the selection of variables .Note that LIMMA inclusion revealed not significant regressors. Thus, LIMMA fitted values do not benefit from regression modeling, under the given hypotheses. This means that by delivering a much wider but different profile than other methods, LIMMA covers the gene space in regions unexplored by the other methods. However, this subspace is uncertain in terms of number of false positives. We cautiously keep LIMMA out of the final average for the fitted values. The LIMMA anomaly was also emphasized with the use of PCA, and in that case too the bias introduced was potentially too high to keep LIMMA’s evidences stacked with the other ones. In terms of final consensus DE profile, we obtained comparative results with crude average of outcome DEGs, total averages from regression, and fitted averages obtained from ARM with and without LIMMA. The most balanced distribution is always obtained in the last scenario.https://github.com/ccs-bio/melanoma-transcriptomics. At this repository, expression tables  and scripts  for both RNA-Seq sample alignment and filtering to extract unique read mappings are reported. R scripts for differential expression can also be found for DESeq, NOISeq, and limma. A bash shell script is also included with the CuffDiff command call, along with the R script for processing the data with CummeRbund. The experimental grouping for GeneSpring is also available., as well as scripts for visualizations. All source code will be released under the GPL v.3 license.All software and datasets can be obtained at the Center for Computational Science’s GitHub repository, available at the following URL: The analysis was based on the Ensembl annotations version 72. Ensembl releases versions each year . The mutational status of BRAF and NRAS is reported in85 too.Cells were grown at 37 °C in a humidified atmosphere with 5% CO2. All the media used for cell culturing are reported inThe cDNA sequences of the genes of interest were obtained from Ensemble, inserting the code available in the tables. In order to align sequences and detect their homology CLC Sequence Viewer 7 was used.http://primer3.ut.ee/), while UCSC in silico PCR (https://genome.ucsc.edu/cgi-bin/hgPc)rwas used to confirm the specificity of primers (absence of off-targets). All primers were purchased from Primm srl . Sequences are reported in Table Primer design was performed using Primer3 , following the manufacturer’s instructions. 1 μg of total RNA was treated with DNAse I, amplification grade  following the manufacturer’s protocol. 0.5 μg of RNA treated with DNAse I was retrotranscribed on a S1000 Thermal Cycler  using iScript cDNA Synthesis Kit . The successful retrotranscription and the absence of contaminating genomic DNA were routinely checked through a control PCR in which exon-spanning primers for ATPA1 gene are used. cDNA was diluted (1:4) and used for qRT-PCR.1 μL of cDNA and appropriate primers (Table Supplementary Files