Developing gene-expression-based classification algorithms is an emerging powerful method for diagnosis, disease classification and monitoring at molecular level, as well as providing potential markers of diseases. Most of the statistical methods proposed for the classification of gene-expression data are either based on a continuous scale (eg. microarray data) or require a normal distribution assumption. Hence, these methods cannot be directly applied to RNA-Seq data since they violate both data structure and distributional assumptions. However, it is possible to apply these algorithms with appropriate modifications to RNA-Seq data. One way is to develop count-based classifiers, such as Poisson linear discriminant analysis and negative binomial linear discriminant analysis. Another way is to bring the data closer to microarrays and apply microarray-based classifiers. In this study, we compared several classifiers including PLDA with and without power transformation, NBLDA, single SVM, bagging SVM (bagSVM), classification and regression trees (CART), and random forests (RF). We also examined the effect of several parameters such as overdispersion, sample size, number of genes, number of classes, differential-expression rate, and the transformation method on model performances. A comprehensive simulation study is conducted and the results are compared with the results of two miRNA and two mRNA experimental datasets. The results revealed that increasing the sample size, differential-expression rate and decreasing the dispersion parameter and number of groups lead to an increase in classification accuracy. Similar with differential-expression studies, the classification of RNA-Seq data requires careful attention when handling data overdispersion. We conclude that, as a count-based classifier, the power transformed PLDA and, as a microarray-based classifier, vst or rlog transformed RF and SVM classifiers may be a good choice for classification. An R/BIOCONDUCTOR package, MLSeq, is freely available at  Transcriptome sequencing (RNA-Seq), with the advent of high-throughput NGS technologies, has become a popular experimental approach for generating a comprehensive catalog of protein-coding genes and non-coding RNAs and examining the transcriptional activity of genomes. Furthermore, RNA-Seq is a promising tool with a remarkably wide range of applications such that (i) discovering novel transcripts, (ii) detecting/quantifying the spliced isoforms, (iii) fusion detection and (iv) revealing sequence variations  . In addiMicroarray-based gene-expression classification has been widely used during the last decades. Recently, RNA-Seq replaced microarrays as the technology of choice in quantifying gene expression due to certain advantages such as providing less noisy data, detecting novel transcripts and isoforms or not requiring prearranged transcripts of interest –5. AlthoSeveral alternatives have been proposed for the classification and clustering of RNA-Seq data. One alternative, perhaps the preferable option,is to use discrete probability distributions  for both classification and clustering tasks. Witten et al.  proposedIn this study, we applied algorithms based on both discrete and continuous distributions to RNA-Seq data. The NBLDA and PLDA are appliedto discrete gene-expression data , while support vector machines (SVM), bagging support vector machines (bagSVM), random forests (RF) and classification and regression trees (CART) are applied to transformed gene-expression data. A comprehensive simulation study is conducted to measure the effect of several parameters on model performances, such as overdispersion, sample size, number of genes, number of classes, DE rate and the transformation method. Four publicly available gene-expression datasetswere also analyzed and the results were compared to the simulation results. An R/BIOCONDUCTOR package, called MLSeq, is developed to analyze RNA-Seq data using the proposed algorithms in this paper.http://www.bioinformatics.babraham.ac.uk/projects/fastqc/), HTSeq [http://edwards.sdsu.edu/cgi-bin/prinseq/prinseq.cgi), FASTX Toolkit (http://hannonlab.cshl.edu/fastx_toolkit/) and QTrim [Providing a comprehensive and easy-to-understand workflow for RNA-Seq studies and its related algorithms helps researchers to find out the background of such studies. We outlined the count-based classification pipeline for RNA-Seq data in ), HTSeq , R Short), HTSeq , PRINSEQnd QTrim  are avaind QTrim . Hence, nd QTrim , MapSplind QTrim  or Star nd QTrim  can be pnd QTrim , bedtoolnd QTrim  and Featnd QTrim . The mapnd QTrim , trimmednd QTrim , reads pnd QTrim  and quannd QTrim  methods.nd QTrim  and NBLDnd QTrim . Secondlnd QTrim , rlog [1nd QTrim  and voomnd QTrim . Apart fnd QTrim .The crucial steps of classification can be considered as feature selection, model fitting and validation. RNA-Seq data has a large dimensions, i.e there are a large number of genes in the count data. In the feature selection step, we aim to work with an optimal subset of data in order to overcome the curse of dimensionality. This process is also crucial to reduce the computational cost, decrease noise, improve the accuracy for classification of phenotypes and work with more interpretable features to better understand the domain . As the k-fold cross-validation, leave-one-out cross-validation and bootsrapping are among the recommended approaches for model validation.In many real life problems, it is possible to find classification algorithms which perform well and perfectly classify training samples. However, it may perform poorly when classifying new samples. This problem is called overfitting/overestimation as previously mentioned. This problem arises during the model training steps. In order to overcome overfitting problem and obtain generalized results, the appropriate model is selected by validating the selected model on independent test samples. Holdout, gj is the total number of mapped counts per gene , si is the number of mapped counts per sample, dkj is the differential-expression parameter of the jth gene between classes k and φ is the dispersion parameter. The datasets contain all possible combinations of the following:A comprehensive simulation study is conducted to investigate the effect of several parameters. Gene-expression data are simulated under 864 different scenarios using a negative binomial model as follows:φ = 0.01 (very slightly overdispersed), φ = 0.1 , and φ = 1 (highly overdispersed);different dispersion parameters such as n) changing as 40, 60, 80, 100;number of biological samples (p’) as 25, 50, 75, 100;number of differentially-expressed genes (dkj) 1%, 5% and 10%;differentially expressed gene rates as (k) as 2, 3 and 4;number of classes  and test sets (30%). All the model building processes are performed on training datasets and model performances are evaluated in test sets.All datasets are initially simulated for xij, the mapped read counts to jth gene for ith sample. Size factor of the ith sample  dataset is an RNA-Seq provided by The Cancer Genome Atlas (TCGA) . The TCGLung cancer is another RNA-Seq dataset provided by the TCGA platform. This dataset contains the read counts of 20,531 transcripts of 1,128 samples. Samples are separated into two distinct subclasses. These subclasses are lung adenocarcinoma (LUAD) and lung squamous cell with carcinoma (LUSC) with 576 and 552 class sizes, respectively. These two classes are used as class labels in our analysis.Real datasets are analyzed using similar procedures to those in the simulation study. Model building is performed on the training set (70%) and the test set (30%) is used to evaluate model performance. Size factors and dispersion functions are estimated for training datasets. Similar to the simulation experiments, the size factors and dispersion functions of test datasets are directly estimated from the training data to make them in the same scale and homoscedastic to each other. Near-zero variance filtering is applied to the training set. Filtered genes are also removed from the test set. The renal cell and lung cancer datasets include 20,531 features which dramatically increase the computational cost. Hence, we initially selected 5,000 genes with the highest variances to eliminate the effect of non-informative mRNAs and decrease the computational cost. All of the miRNAs are used in the model building process for the cervical and Alzheimer datasets. Differential expression was performed on the training data using the DESeq2 method and genes are ranked from the most significant to the least significant with increasing number of genes in steps of 25 up to 250 genes. The differentially expressed genes selected in the training data are also selected in the test datasets. Differentially expressed genes in the training data are normalized using the median ratio approach and transformed using either the vst or rlog approaches. Since the sample sizes of the cervical and Alzheimer miRNA datasets are relatively small, the entire process is applied 50 times. The other model building processes applied are similar tothose in the simulation study.1), Poisson linear discriminant analysis with power transformaton (PLDA2) and negative binomial linear discriminant analysis (NBLDA). In this section, we will summarize the background and use of each method.Both simulated and real data are modeled using support vector machines (SVM), bagging support vector machines (bagSVM), random forests (RF), classification and regression trees (CART), Poisson linear discriminant analysis without power transformaton  = Φ(xi)Φ(xj)) and the linear classification then applied in this high-dimensional space. Some of the most widely used kernel functions are linear: K = xixj, polynomial: K = (xixj + 1)d, radial basis function: K = exp (−γ‖xi−xj‖2) and sigmoidal: K = tanh(k(xixj) − c) where c is a constant, d is the degree and γ > 0 is sometimes parametrized as γ = 1/2σ2. Normalized and transformed (either using vst or rlog) datasets are used as input to the SVM classifier. The radial basis kernel function is used in the analysis.In a binary classification, the SVM aims to find an optimal separating hyperplane in the feature space which maximizes the margin and minimizes the misclassification rate by choosing the optimum value of  b in Eq . When thd Vapnik  can be uk random bootstrap samples are drawn with replacement. SVM classifiers are trained independently on each randomly selected subsets and aggregated via an aggregation technique.A test set is predicted on each of the SVM classifiers and the predicted class labels are determined using aggregated results likely in training sets. Normalized and transformed datasets are used as input to the bagSVM classifier. The number of bootstrap samples were set to 101 since small changes were observed over this number.bagSVM is a bootstrap ensemble extension of SVM which creates individuals for its ensemble by training each SVM classifier on a random subset of the training set. For a given data set, p(i|j) is the probability of class i at node j, the Gini index is calculated using the equation 1 − ∑ip2(i|j). It is possible to obtain very large CART trees in large data sets, i.e very large number of genes and samples. When CART grows a maximal tree, this tree is pruned upward to get a decreasing sequence of subtrees. Furthermore, pruning is preferred to overcome overfitting problem. The optimal tree that has the lowest misclassification rate is selected using a cross-validation.The assignment of each terminal node to a class is performed by choosing the class that minimizes the resubstitution estimate of the misclassification probability [CART, which was introduced by Breiman , is one bability , 38. Norm features, for example, mtry out of m features is randomly selected at each node while growing the forest. Different splitting criterias can be used such as the Gini index, information gain and node impurity. The value of mtry is approximately equal to either An RF is a collection of many CART trees combined by averaging the predictions of individual trees in the forest . RF aimsll trees , 41. NorX be an n × p matrix of the sequencing data where n is the number of observations and p is the number of features. For sequencing data, Xij indicates the total number of reads mapping to gene j in observation i. The observed counts are fitted to the Poisson log-linear model as given in Eq , i = 1,…,n be a training set and x* is assigned to one of the classes with highest probability(or discrimination score) using the Bayes’ rule as follows:y* denotes the unknown class label, fk is the probability density of an observation in class k and πk is the prior probability that an observation belongs to class k. If fk is a Gaussian density with a class-specific mean and common variance, then a standard LDA is used to assign a new observation to the class [yi is the class of the ith observation and the features are independent. Eq  is applied and the sparsest model with the highest accuracy rates is selected for classification.Normalized count data are used as input to the PLDAφ) of negative binomial distribution and called the method negative binomial linear discriminant analysis (NBLDA). This extra dispersion parameter is estimated using a shrinkage approach detailed in [Dong et al.  generaliailed in . A new tAs the dispersion decreases, NBLDA approximates to PLDA. More details on NBLDA can be found in .TP + TN)/n based on the confusion matrices of test set class labels and test set predictions. For multiclass scenarios, these measures are calculated via the one-versus-all approach. Since, the class sizes are unbalanced in the Alzheimer and renal cell cancer datasets, accuracies are balanced using the following formula: (Sensitivity + Specificity)/2.To validate each classifier model, five-fold cross-validation is used. It is repeated 10 times and accuracy rates are calculated to evaluate the performance of each model. Cross-validation folds are fixed for all classifiers to make the results comparable to each other. Accuracy rates are calculated as , while the lung and renal cell cancer datasets are substantially overdispersed. The simulation results for k = 2 and k = 3, dkj = 10% for rlog transformations are given in http://www.biosoft.hacettepe.edu.tr/MLSeqSupplementary/ and the Genewise dispersion parameters are estimated for each classifier usingthe method of moments. Distribution of the estimated overdispersions is given in dkj = 10%. Likewise, classification accuracies slightly increase in real datasets since increasing the number of genes leads to an increase in the probability of a differentially expressed gene being included in the classification model. However, this may not be true for all cases. Since, the classification accuracy may decrease from certain point as more genes were included in the model. Since, increasing the number of genes will lead to an increase in the model complexity, we expect that the classification error of test samples will be reduced. For the PLDA classifier, a high number of selected genes provides alternative options for the lasso shrinkage method to test more genes in classification models. The RF algorithm builds trees with the bagging approach. As the number of genes increases, the RF algorithm uses more genes and may perform better to specify the optimal tree. Increasing the sample size, on the other hand, improves the discrimination power and classification accuracy. Conversely, overall accuracies decrease as the number of classes increases. This is due to the fact that the probability of misclassifying an observation may arise depending on the complexity of the model. As the number of classes increases, the samples become less likely to be linearly (or non-linearly) separable.Since combining each significant gene on class conditions is equivalent to combining their predictive abilities, an increased number of differentially expressed genes leads to an increase in the classification accuracy . Similark = 2 and dkj = 10%. As the data become more spread out , the variance increases; thus, we need more samples to better learn from data and achieve the same discrimination power. When we stabilize the sample size and increase the dispersion parameter, this leads to a decrease in the discrimination power and classification accuracies. Nagalakshmi et al. [The overall performance of each model increases as the data become less dispersed. Decreasing the dispersion parameter makes a significant contribution to classification accuracy, as expected, even for the same data and the same scenario. This is more obvious when i et al.  reportedi et al. , 45–46. i et al. . This ovi et al. , 46–49. p and small n settings. However, technological improvements make RNA-Seqa state-of-the-art approach for quantified transcriptomics. Currently, much of these microarray-based classifiers cannot be directly applied to RNA-Seq data because of the discrete nature of RNA-Seq data. Microarray data consist of the continuous log-intensities of probes while RNA-Seq data consist of the discrete and overdispersed mapped read counts of sequencing technologies. The results of this study revealed that, transforming the RNA-Seq data and bringing them hierarchically closer tomicroarrays (e.g. through rlog and vst) might be a suitable approach to make the microarray-based classifier applicable for RNA-Seq data.Hundreds of microarray-based classifiers have been developed and are able to work in large Witten et al.  stated t1 and PLDA2 show similar performances  can be efficiently used for the classification of RNA-Seq data.Besides RNA-Seq’s advantages over microarrays, the gene-expression data from RNA-Seq are overdispersed due to inherent variability. This overdispersion seemed to be a drawback for differential expression studies of RNA-Seq data. In this study, we showed that this overdispersion is also a drawback for classification studies since an increase in the variance will lead to a decrease in the discrimination power. We reached the conclusion that three solutions are available to handle the classification of overdispersed RNA-Seq data: (i) increasing the sample size, (ii) transforming the data to bring RNA-Seq data hierarchically closer to microarrays usingvariance stabilizers, e.g. vst and rlog transformations and (iii) using count-based classifiers such as the PLDATo make an overall assessment for the performances of classifiers, the PLDA after a power transformation may be a good choice as a count-based classifier. Furthermore, its sparsity seems to be an advantage for researchers. However, further studies are needed. Surprisingly, the performance of the NBLDA was not satisfactory as a count-based classifier. Dong et al.  reportedMoreover, an alternative option may be to bring the data closer to microarrays and use microarray-based classifiers. Our results revealed that the RF, SVM and bagSVM may give accurate results after an rlog or vst transformation. Moreover, the efficiency of the bagSVM is improved markedly with increasing sample size.https://www.bioconductor.org/packages/release/bioc/html/MLSeq.html.We conclude that data with less overdispersion, highly differentially expressed genes, a lower number of groups and large sample size may improve the accuracy of the classifiers. Finally, we developed an R/BIOCONDUCTOR package, called MLSeq, for the classification of RNA-Seq data. This package can be accessed and downloaded through S1 File(ZIP)Click here for additional data file.S2 File(ZIP)Click here for additional data file.S3 File(ZIP)Click here for additional data file.S4 File(DOCX)Click here for additional data file.S5 File(DOCX)Click here for additional data file.


It is interesting to study the consistency of outcomes arising from two genomic platforms: Microarray and RNAseq, which are established on fundamentally different technologies. This topic has been frequently discussed from the prospect of comparing differentially expressed genes (DEGs). In this study, we explore the inter-platform concordance between microarray and RNASeq in their ability to classify samples based on genomic information. We use a set of 7 standard multi-class classifiers and an adaptive ensemble classifier developed around them to predict Chemical Modes of Actions (MOA) of data profiled by microarray and RNASeq platforms from Rat Liver samples exposed to a variety of chemical compounds. We study the concordance between microarray and RNASeq data in various forms, based on classifier’s performance between two platforms.Using an ensemble classifier we observe improved prediction performance compared to a set of standard classifiers. We discover a clear concordance between each individual classifier’s performances in two genomic platforms. Additionally, we identify a set of important genes those specifies MOAs, by focusing on their impact on the classification and later we find that some of these top genes have direct associations with the presence of toxic compounds in the liver.Overall there appears to be fair amount of concordance between the two platforms as far as classification is concerned. We observe widely different classification performances among individual classifiers, which reflect the unreliability of restricting to a single classifier in the case of high dimensional classification problems.Camda Satellite Meeting to Ismb 2015 by the Camda Programme Committee. The full research paper then underwent two rounds of Open Peer Review under a responsible Camda Programme Committee member, Lan Hu, PhD . Open Peer Review was provided by Yiyi Liu and Partha Dey. The Reviewer Comments section shows the full reviews and author responses.An extended abstract of this research paper was selected for the  For more than a decade microarray technology has provided enormous momentum to the modern genomic research. The ability of quantify thousands of genes’ expressions at the same time has led to remarkable achievements in wide range biological studies. Abundance of microarray assays has been published worldwide in various databases. However, microarray technology has some limitations, such as the accuracy of expression measurements limited by levels of hybridization and variability hybridization properties of probes . RNAseq In recent years, a number of authors have discussed the agreement between scientific conclusions made on microarray and RNASeq platforms, based on comparative analyses. A common choice for these studies was the concordance of differentially expressed genes (DEGs). A previous study that described a large scale comparison of microarray and RNASeq platforms using the Cancer Genome Atlas (TCGA) based analysis, reported a high correlation among expressions levels resulted from both platforms and suggested a reasonable concordance between DEGs by comparing tumors with normal tissues . AnotherOur work is based on the previously described rat liver data , where wThe remainder of the article is organized as follows. In Section “We first describe outcomes of the Analysis 1, which was performed using two basic strategies: adjusted and originally given test sets described in Section “r=0.92). The accuracy of individual classifier varies from 17 to 75%, and as to be expected, the performance of the ensemble classifier is the best in both platforms. The overall accuracy of the optimal classification method is slightly better in microarray compared to RNA-seq (75% vs 67%). In particular, we observe a lower prediction accuracy for the class “PPARA" in RNASeq (56%), compared to the microarray (89%) platform. Overall, the class given by “CAR/PXR" which has a maximum sensitivity of only 56%, seems to be the MOA that hardest to predict. Some individual classifiers show widely different prediction sensitivity for the same class in two platforms. For example the sensitivity for “PPARA" by RPART is 100% in microarray, whereas it reaches as low as 22% in RNAseq.We first discuss the classification resulted from using a set of genes that are represented in both platforms. For the adjusted test set, the left panel of the Fig. r=0.94) as shown in the right panel of the Fig. When the original  test set is used, we again observe matching performance of classifiers in both platforms Table  similar Even with the complete set of genes, we observe similar conformity of classifiers’ performance between the two platforms Fig.  as descrr=0.99). Remarkably, the ensemble classifier was able to provide 100% accurate predictions for both cases, regardless of the additional complexity caused by 8 varieties of classes. In this analysis, the component classifier PLS+LDA also performed similarly to the ensemble classifier in both cases yielding 100% accurate class predictions. Apart from above two classifiers, SVM, RF, and PLS+RF also hold substantially high prediction accuracies.Results of the 2nd analysis, namely, classification in cross platform are summarized in Table Exploring outcomes resulted from Analysis 1 and 2 Tables , 4 and 5We summarize results of the 3rd analysis in Tables Referring to the Table Table R), for microarray and RNASeq, respectively. Clearly, there is no single gene that makes a substantial contribution to the accuracy. However, we identified two genes  that are commonly present in both lists when the complete set of genes was used. Based on the same analysis but performed using complete sets of genes we observe only one gene named Id1 is common important gene for the two platforms. We observed that Abcc3 is a member of the superfamily of ATP-binding cassette (ABC) transporters, which is involved in multi-drug resistance [Although the main goal of detecting impotent genes with the full data  was to identify sets of genes making considerable impact on classifying all eight MOAs, interestingly, the outcome of this study Tables  and 9 resistance . The Id1sistance .Support Vector Machines (SVM), Random Forests (RF), Neural Network (NN), Linear and Quadric Discriminant Analysis  are examples of standard techniques that are widely applied in classification problems. Performances of these classifiers are highly variable across problems. Thus, none of standard classifier can be considered to be the best for all classification settings. In complex situations, such as classifications in high dimensional genomic data, a more meaningful approach would be use an ensemble classifier which combines many standard classification algorithms together to develop an improved classifier. The ensemble classifier we use builds a number of individual models on randomly selected subsets of data which can then be combined or averaged in some meaningful fashion. Majority voting is a popular choice is for a typical solution. Such a classifier by allowing data based utilization of a multitude of classification algorithms for a upholds consistent performance in various types of data and classification problems. In this work, we use the adaptive optimal ensemble classer developed, via bagging and rank aggregation . In thisn samples with p dimensional covariates in the form of { Xn×p,Yn×1}, where X corresponds to independent variables and Y represents the dependent categorical variable that specifies a class label. Assume the ensemble classier is intend to built with M classification algorithms based on K different performance methods such as overall accuracy, class sensitivities etc. to optimize the predictive performance. Thus, we proceed as follows: Resampling: Draw a bootstrap sample of size nXn×p,Yn×1} by resampling rows with simple random sampling. Sampling is repeated until samples from all classes are present in the bootstrap sample and then determine the corresponding OOB sample that contains all samples which are left out from the bootstrap sample.Classifier Training: Train M classification algorithms, C1,…,CM, on the bootstrap sample.Performance Assessment: Obtain M predicted class labels for each OOB case. Since true classes of the OOB samples are known, calculate K different performance measures for each of M algorithms using their corresponding predictions.Rank Aggregation: Rank M algorithms according to K performance measures. So, we have K ordered lists  of size M. These lists are then rank-aggregated using the weighted rank aggregation to determines the best algorithm C(1) overall.B times, where B considered to be a large integer which is usually selected according to the computational capacity.Repeat the above procedure (steps 1–4) for Prediction for a New Sample: Predict the class variable Y for a new sample X using the B prediction models The algorithm described below demonstrates the step by step procedure of developing an ensemble classifier . SupposeM classifiers are evaluated on the basis of K performance measures. Assume we have ordered lists L1,…,LK, where ith ordered list Li, i=1,…K, provides ranks of M algorithms on their performances evaluated on the ith measure. The rank aggregation [M classifiers that minimizes the weighted sum of distances from all individual lists, given by the following objective function, Suppose the performances of regation , 16 procL is any possible ordered list of the M classifiers, wi’s are weights which represent the user specific importance of each of the K performance measures. The classifier in the first position of this aggregated list that is the optimal classifier overall with respect to all the validation measures. Of course, the default choice would be to use wi=1 for all i which means all the validation measures are taken as equally important in determining the optimal algorithm. Throughout out analyses, we have used wi=1. d is a distance function such as Spearman’s footrule or Kendall’s tau, which measures the closeness between two ordered lists. In this work, we use Spearman’s footrule distance function as the distance measure.where L1 regularization, etc., for greater applicability and improved prediction accuracy [Often for high dimensional data, standard classifiers are combined with dimension reduction, variable selection, or penalization techniques such as Partial Least Squares (PLS), Principle Component Analysis (PCA), Random Forest (RF) based importance measures, accuracy , 19. ForOur data for this study was released by 2015 CAMDA competition. Microarray and RNASeq platforms contain gene expression measurements of nearly 31,000 and 46,000 genes, respectively. The dataset consists of gene expression responses profiled by Affymetrix microarrays and Illumina RNASeq sequencer in rat liver tissues from 105 male Sprague-Dawley Rats, which are exposed to 27 different chemicals represented by 9 different MOAs. In the original experiment, a training set is formed with 45 rats, which are treated with 15 chemicals corresponding to MOAs of “PPARA", “CAR/PXR", “AhR", “Cytotoxic", “DNA damage", and 18 controls. Test set contains data on 36 rats which are treated with 12 chemicals corresponding to “PPARA", “CAR/PXR", “ER", “HMGCOA" and 6 controls. We found that two MOAs, “ER" and “HMGCOA" are present only in the test set. We further noticed that approximately 22,253 average expressions per sample in RNA-seq data were recorded as “NA", which indicates that insufficient number of reads mapped onto the gene to provide a reliable gene expression estimate. We retained gene sets of sizes 13,686 and 16,133 for microarray and RNASeq platforms, after (i) removing unnamed genes, (ii) removing genes with unobserved expressions, and (iii) averaging multiple expressions reported from the genes with unique names.In this work, we used normalized expression levels that came from microarray data using Robust Multi-Array Average (RMA) expression measurements , whereasTrain classifiers and make predictions on individual platforms.Train classifiers in one platform to make predictions on the other platform.Identify important variables (genes) for accurate classification.We conducted three types of investigations for studying the performance of the proposed classifiers. B bootstrap samples.In the 1st analysis, we explore the predictability of MOAs using various classifiers developed in the given training data. To our knowledge, there is no established criteria to define prediction for an unknown class that was not represented in the training data. Thus, we select an adjusted test set after eliminating all test samples belonging to two classes of “ER" and “HMGCOA", where the new test was used in parts of 1st and 3rd analysis. However we also considered the originally given test set as a part of 1st analysis by adopting following alternative classification approach. Accordingly, first we designated both “ER" and “HMGCOA" samples belonging to the original test set as “OTHER". For each classifier, then we determined the maximum class probability for a given test sample and if the above probability was less than 0.5 we selected the predicted class as “OTHER", else kept the originally predicted class. For this purpose, class probabilities for the ensemble classifier was calculated using the predicted class proportions observed in the Our objective with the 2nd analysis was to examine the inter-platform concordance between microarray and RNAseq platforms. Thus, we trained classifiers on a selected platform using the full dataset that included the both given training and test sets for making predictions on the other platform. However, since the classifier needed to run on both platforms for this analysis, each gene expression measurement was standardized, separately for both platforms, prior to the analysis.M=7 standard classifiers, SVM, RF, LDA, PLS+RF, PLS+LDA, PCA+RF, PCA+LDA, and Recursive Partitioning (RPART). Primarily, classifiers are selected based on the prior information of their suitabilities in high dimensional data classification. Based on accuracies of predicted classes, each classifier was ranked for K number of performance measures . Since the selection of performance measures for a multi-class classification problem is highly depend upon the aim of study; we optimized the overall prediction accuracy, and the class specific accuracy of each group for the 1st analysis. Furthermore we considered these performance measures to be equally important for classification , such that the rank 1 corresponds to the gene that has the highest negative impact on the overall prediction accuracy. In order to reduce the computational burden, we did not use the ensemble classifier for this purpose. Instead the component classifier PLS+LDA which had an overall accuracy close to that of the ensemble classifier was used. We performed theses analysis separately for both platforms to determine a common set of genes presented among the top 20 genes in both platforms.l times for each gene to calculate an average overall prediction accuracy (A). Finally, genes were ordered by A, ascending order. Here we chose l to be l=30 in order to achieve reasonably stable approximation, while keeping the computational costs in check.For Analysis 3.1, we randomly permuted a gene’s expressions in the training set and then made predictions for the test set (adjusted test set) using the classifier trained on the permuted training data. The permutation procedure was repeated jth, j=1,...,J, outer randomization, we randomly partitioned the dataset into 5 folds and selected a training set of 4 folds, while remaining fold was chosen as a test set. After randomly permuting the expressions of a given gene i across the above specified training set, a classifier was trained to predict on the selected test set. Now using the same approach we described in the previous part  we obtained an average overall prediction accuracy l times. After that, the whole procedure was repeated J times for various random partition sets to obtain an average overall prediction accuracy  based on J random partition sets. Note that the magnitude of Acv can be varied. Thus a better measure will be a relative accuracy reduction (Ri) given by, Ri indicate high impacts on the classification. For Analysis 3.2, we used values l=30 and J=100, which stabilize the calculations without being computationally burdensome.Suppose In this study, we used an ensemble classifier built on a set of standard classifiers to predict the MOA in Rat liver experiment data profiled by both microarrays and RNASeq. The newly constructed ensemble classifier performed reasonably well in both platforms individually. Using a selected test set and a set of genes (those present in both platforms) we observe comparable overall predictability of MOAs in the two platforms with 75% and 67% accuracies for microarray and RNAseq, respectively. Similarly, we observe well matched accuracies of 50% for both platforms for the full test sets based on an alternative approach. In an earlier classification approach [In this study, we explored the inter-platform concordance between microarray and RNASeq in their ability to classify samples based on genomic information, using data profiled by a Rat Liver experiment. We used an ensemble classifier built on a set of seven standard classifiers to predict the MOA in Rat livers. The ensemble classifier performed reasonably well in both platforms individually, resulting respective 75% and 67% accuracies for microarray and RNAseq on a selected test set. When we trained classifiers to make predictions on cross platforms, the ensemble classifier provided remarkable 100% accurate predictions. This study demonstrates a high agreement between individual classifiers’ performances in two genomic platforms. Additionally, we identified a set of important genes those specifies MOAs, by focusing on their impact on the classification.wi’s) they used in rank aggregation of the ensemble classifier. The authors explained the main idea of the aggregation method, but explicitly stating all the parameters could improve the readability of the paper.I suggest the authors add descriptions on the weights  on Rat Liver samples. The article is well written and nicely organized. In addition, addressing these few points should increase the impact of the research work across various spectrum of readers: Sampling with replacement is part of bootstrap which is standard procedure in bagging. An advantage is that training data of the same size as the original can be created and also the out of bag samples can be used as test data in a natural way. The classifiers were described in Datta et al. (2010). The tuning parameters were selected by cross validation as described in the paper. The ensemble classifier is optimal in an overall sense as defined by the rank aggregation procedure. It may not be possible to beat every individual classifier with respect to every performance measure. We have commented on some genes in Section “”.Authors’ response:


In the FANTOM5 project, transcription initiation events across the human and mouse genomes were mapped at a single base-pair resolution and their frequencies were monitored by CAGE  coupled with single-molecule sequencing. Approximately three thousands of samples, consisting of a variety of primary cells, tissues, cell lines, and time series samples during cell activation and development, were subjected to a uniform pipeline of CAGE data production. The analysis pipeline started by measuring RNA extracts to assess their quality, and continued to CAGE library production by using a robotic or a manual workflow, single molecule sequencing, and computational processing to generate frequencies of transcription initiation. Resulting data represents the consequence of transcriptional regulation in each analyzed state of mammalian cells. Non-overlapping peaks over the CAGE profiles, approximately 200,000 and 150,000 peaks for the human and mouse genomes, were identified and annotated to provide precise location of known promoters as well as novel ones, and to quantify their activities. The initial key technology driving the project was to make full-length cDNA clones, representing complete primary structure of transcribed RNA molecules. Sequencing of the full-length cDNA clones uncovered unexpected number of long non-coding RNAs as well as protein coding genes2–6. The CAGE 8 protocol, combination with high-throughput sequencing, was developed to monitor frequencies of transcription initiation by determining 5′-end of capped RNAs. The technology was devised to uncover complexity of the transcriptome4–6 and elucidate transcriptional regulatory networks by focusing on promoter elements9–12. By taking advantage of single molecule sequencer, HeliScopeCAGE was recently developed to provide more sensitive and accurate monitoring of transcription initiation activities8.Since the completion of the human genome sequencing, role of individual bases has been a central question. An international collaborative effort, FANTOM 13. Our central focus was on human primary cells, while cell lines, tissues and mouse samples were chosen to cover cells inaccessible as isolated human primary samples. The resulting data provided an atlas of promoter and enhancer activities in wide range of cell states14, which is a baseline of understanding complex transcriptional regulation. In the second phase, we focused on transitions of cell states by monitoring ‘time course’ samples, such as activations, differentiations, and developments at sequential time points15. The monitored activities of promoters and enhancers demonstrated that enhancer activities is the earliest event during dynamic changes of transcriptome. These data sets are being utilized in many other studies inside and outside of the FANTOM5 consortium.In the fifth round of the FANTOM projects, FANTOM5, the challenge was to capture the transcriptome of many varieties of cell states as possible, to understand the implication of each genomic bases in different contexts. In the first phase of the FANTOM5 project, we targeted cells in steady state, called ‘snapshot’ samples8 and a robotic workflow for samples with standard requirements7. The assay pipeline started with checking RNA quality , which provides a uniform quality assessment of the profiled RNA extracts. The resulting CAGE libraries were sequenced by HeliScope in RIKEN and also in Helicos Biosciences, and the obtained data were processed by the MOIRAI system16. Quality of the resulting CAGE profiles was checked with several statistics as well as manual inspection by using the ZENBU browser17. Finally CAGE profiles were shared among the consortium for further analysis.The data production scheme was implemented based on the FANTOM5 collaboration. Sample collection was performed at individual institutes, since specific types of samples require dedicated systems with special expertise or settings, as well as through purchase from commercial sources. RNA quality was firstly examined at the place where the samples were obtained . The CAGE assay pipeline established in RIKEN GeNAS  employed two workflows of HeliScopeCAGE, a manual workflow for samples with small amount of total RNAshttp://fantom.gsc.riken.jp/5/)18. Here we describe the data with the processing details and quality metrics.In the course of the two phases focused on ‘snapshot’ and ‘time course’ samples, we profiled 1,816 human and 1,016 mouse samples in total, and obtained approximately four millions of single-molecule reads successfully aligned to the genome per sample on average. Based on frequencies of the observed 5′-ends of individual capped RNA molecules at a single base-pair resolution, we identified 201,802 and 158,966 peaks for human and mouse respectively, where promoters are defined as the sequence immediately upstream of the peaks and frequencies of observed CAGE reads reflect activities of the promoters. All data generated during the course of the project were deposited to a public repository  and/or provided at the FANTOM5 web resource  was used for RNA extraction. Human post mortem tissue RNAs were purchased or obtained through the Dutch Brain bank. Tissues collected through the consortium were snap-frozen in liquid nitrogen, transferred into Lysing Matrix D tubes  containing chilled Trizol (Gibco), homogenized by FastPrep Homogenizer (Thermo Savant), and centrifuged. miRNeasy kit (QIAGEN) was used for RNA extraction from cultured cell lines as well as frozen cell line stocks.Sample collection was performed as described previouslyFor the purchased samples, lot or catalogue numbers were recorded where available. Of the collected RNAs, those with more than 1 μg, were measured by Agilent BioAnalyzer  and Nanodrop spectrophotometer  to check RIN (RNA integrity) score and the absorbance ratio of A260/A230 and A260/A280. The rest of the samples were directly subjected to the CAGE library production to avoid wasting material. All 2,832 profiled samples are summarized in 15. Most of the RNAs were subjected to the automated HeliScopeCAGE protocol7, except for RNAs with less than 1 μg that were subjected to the manual protocol optimized for low quantity RNAs8. The resulting libraries were measured by OliGreen fluorescence assay kit (Life Technologies), and sequenced by following the manufacturer’s instructions , remaining reads were aligned to the reference genome of human and mouse (hg19 or mm9) by using Delve25, and alignments with a quality of less than 20 (<99% chance of true) or a sequence identity of less than 85% were discarded. Frequencies of the CAGE read 5′ ends were counted to give a unit of CAGE tag start site (CTSS), a single base-pair on the reference genome. The entire flow of the data is illustrated in The produced data were processed as previously describedhttps://github.com/hkawaji/dpi1/) method and annotated as previously described15. A ‘robust’ threshold, for which a peak must include a CTSS with more than 10 read counts and 1 TPM (tags per million) at least one sample, was employed to define a stringent subset of the CAGE peaks. The robust peaks were associated with known transcripts, such as RefSeq26, UCSC known gene27, GENCODE28, Ensembl29, and mRNAs (full-length cDNA clones), based on their 5′-end proximity to the peaks. Official gene symbols, Entrez Gene IDs, and protein (UniProt) IDs associated with the transcripts were retrieved and assigned as part of annotation. In addition to these associations, human readable names and descriptions were assigned to each of the CAGE peaks. Peaks were given a name in the form pN@GENE, where GENE indicates gene symbol or transcript name and N indicates the rank in the ranked list of promoter activities for that gene. For example, p1@SPI1 represent the peak with the highest number of observation  in all of the FANTOM5 CAGE profiles, among the peaks associated with SPI1 gene.Non-overlapping peaks based on the all CAGE profiles were identified by using DPI , and the second was for the entire samples from both the ‘snapshot’ and ‘time course’ studies (phase 2). We integrated these two peak sets into a hybrid set consisting of all the phase 1 peaks over the robust threshold and a subset of phase 2 peaks that did not overlap with the phase 1 peaks. Annotation of phase1 peaks was used in the hybrid set, called phase 1+2 peaks, which provide a consistent reference in the definition of promoters.30 method by edgeR31, and tags per million  was computed as expression levels.All the obtained CAGE profiles were subjected to the peak identification, even if they have some issues in quality, since all of them still represent independent observations of RNA 5′-ends. However promoter activities  were quantified only in the samples satisfying the following criteria: RIN score greater than 6, more than 500,000 successfully aligned reads to the genome, and more than 50% of the successful alignments are close to 5′-end of RefSeq gene model, for expression analysis requiring reliable quantification. After discarding a few CAGE profiles of low quality, read counts for individual CTSSs belonging to the same peak were summed up, normalization  factors were calculated with RLE (Relative Log Expression)15.The RLE normalization was first performed within the phase 1 samples. The naïve application of this to the entire data sets, consisting of phase 1 and phase 2 samples, might cause inconsistencies in expression levels between the two normalizations. To avoid this, we took the geometric mean of CAGE peak read counts across the phase 1 samples and used it as the reference expression for a normalization factor calculation in the same manner as RLE method. This enabled us to keep the expression levels of phase 1 as they were, and to adjust the expression levels of the phase 2 samples to be comparablehttp://fantom.gsc.riken.jp/5/suppl/rRNAdust/. Mapping software Delve is available at http://fantom.gsc.riken.jp/5/suppl/delve/. The program to perform DPI, decomposition-based peak identification, method is available at https://github.com/hkawaji/dpi1/.All software used in this study are publicly available. rRNAdust, for removing ribosomal RNA, is available at 32. Sample metadata for human and mouse are ‘HumanSamples2.0.sdrf.xlsx’ and ‘MouseSamples2.0.sdrf.xlsx’, respectively. The metadata for the CAGE assay are available as ‘*sdrf.txt’.Two types of metadata are available at figshare and LSDB Archive . One is for the samples, including their origins and extracted RNA. The other is for the CAGE assay, including the result of RNA quality check, library production, and post-processing of the CAGE tag sequences. Both of them are described in SDRF (Sample and Data Relationship Format)All of the CAGE sequences, their alignment to the genomes, and CTSS frequencies are available at DDBJ DRA (DDBJ Sequence Read Archive) (Data Citations 2–9). The accession number of each file is summarized in ‘DRA*.txt’ at figshare (Data Citation 1).https://sourceforge.net/projects/osctf/.Genomic coordinates, annotations and expressions of the CAGE peaks are available as ‘*phase1and2combined_coord.bed.gz’, ‘*phase1and2combined_ann.txt.gz’, and ‘*phase1and2combined_tpm.osc.txt.gz’ respectively at figshare (Data Citation 1). Genomic coordinates are formatted in BED format, and the others are formatted in OSCtable (Order Switchable Column table). The detail of the OSCtable format is available at Measured RNA qualities at the second check  are shown in 13. The average ratio of promoter-hitting reads is 76.5, and 98.6% of the all profiles  have more than 50% promoter-hitting rate, which was another criterion of profiles used for expression analysis13.The number of CAGE reads successfully aligned with the genome and the ratio of CAGE reads hitting conventional promoters are shown in 13 within the phase 1 was shown in 13 was in Hierarchical clustering of the 126 mouse primary cells33. TET (Table Extraction Tool) provides an interface to obtain a subset of data by specifying the desired columns and rows. The BioMart interface34, and FANTOM5 SSTAR  provides the metadata of the profiled samples35. The CAGE profile on the genomic axis is visible in ZENBU17 with its interactive interface and also in the UCSC genome browser36 via track data hub37.As well as providing access to individual data files, we also set up a series of interfaces as described in the FANTOM web resourceHow to cite this article: Noguchi, S. et al. FANTOM5 CAGE profiles of human and mouse samples. Sci. Data 4:170112 doi: 10.1038/sdata.2017.112 (2017).Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


Expression profiles of cancer are generally composed of three dimensions including gene probes, patients  and tissues . In order to combine these three dimensions, we proposed a joint covariate detection that not only considered projections on gene probes and tissues simultaneously, but also concentrated on distinguishing patients into different groups. Due to highly lethal malignancy of hepatocellular carcinoma, we chose data GSE6857 to testify the effectiveness of our method. A bootstrap and accumulation strategy was introduced in, which could select candidate microRNAs to distinguish metastasis from non-metastasis patient group. Two pairs of microRNAs were further selected. Each component of either significant microRNA pair was derived from different cliques. Targets were sought and pathway analysis were made, which might reveal the mechanism of venous metastasis in primary hepatocellular carcinoma. Therefore, identifying molecules that can suppress metastasis may provide novel targets for HCC therapies. MicroRNAs (miRNAs) are a class of highly conserved short RNAs that regulate diverse cellular processes by binding to the 3′untranslated region (3′-UTR) of target messenger RNAs (mRNAs)2. To date, several miRNAs have been characterized to have proangiogenic (miR-2213) or antiangiogenic  activities or to possess prometastatic  or antimetastatic  functions in HCC. Therefore, miRNAs could serve as therapeutic targets in HCC.Globally, hepatocellular carcinoma (HCC) is a common and highly lethal malignancy. It has been generally accepted that the invasive and metastatic potentials of HCC are mostly attributed to rapid recurrence and poor survival of HCC16, virus and non-virus patients17, vascular invasion and primary HCC specimens18, and metastasis and non-metastasis samples20. Besides, a multivariate t-test with permutations of group labels was provided for identification of miRNAs associated with HCC metastasis21. Methods mentioned above were also used to establish gene signatures for HCC metastasis22 or HCC recurrence24.Different miRNAs associated with HCC were derived due to various statistical methods used for screening on genome-wide expression profiles. Note that expression profiles are composed of features or variables (e.g. miRNAs and mRNAs) in row, each of which is across different samples or patients. Commonly, a univariate paired t-test was performed to identify significant miRNAs for discrimination of two groups such as cancer and normal tissues20. However, it has been indicated that two individual features, each of which is differentially expressed, may not correspond to the pair with a best discriminative performance25. Second, most of existing methods treated cancer and adjacent normal tissues separately. As far as HCC metastasis is concerned, statistical analysis was made either only on HCC tissues22 or on HCC and adjacent normal tissues respectively21. A certain combination of cancer and adjacent normal expressions is to be made so that a better discriminative performance of selected features between two groups can be justified. Third, most of feature selection methods were based on hypothesis testing, which aimed to evaluate whether two populations of samples were significantly different or not by a certain discriminative statistics. On the contrary, classification that aimed at classifying samples into the right population they belong to was only viewed as a posterior validation of features selected by anterior hypothesis testing.In fact, these obtained statistical significances are faced with three major problems. First, prevailing studies mainly extracted individual features regardless of their coordination. It was reported that additions or subtractions of expression values from two individually selected miRNAs were provided for a better discriminative performance26 were provided. In view of the limited sample size that led to a poor covariance matrix of each group, we presented an approximate implementation for simplicity. Third, the thought of integrative hypothesis testing (IHT)27 was introduced in. To be specific, we developed Fisher’s LDA-based classification together with Welch’s t-test to be an IHT for coordinative selection of individual or pairwise feature candidates. We implemented the joint covariate detection approach on miRNA expression profiles of primary HCCs publicly available at the gene expression ominbus (GEO) with its accession number GSE685721, and ultimately extracted two miRNA pairs that might be associated with HCC venous metastasis. Potential target genes of these miRNAs were selected using TarBase28 and the corresponding KEGG pathway was selected using DAVID29, which testified the significance of the selected miRNAs.On the basis of these insights, we proposed a joint covariate detection method that combined not only cancer and adjacent normal expression profiles but also hypothesis testing and classification methods. First, individual features and feature tuples on expression profiles were simultaneously taken into account. Considering a large amount enumeration of feature tuples, we only performed up to feature pairs for simplicity. Second, a linear projection on cancer and adjacent normal expressions of each feature was made. Commonly, expressions of each feature were regarded as a basic sampling unit in a vector form. That is, the expressions of each sample on this feature formed a two-dimension vector with its two components representing cancer and normal tissue respectively. Regardless of the correlation among features, Fisher’s linear discriminative analysis (LDA) was made on each feature with corresponding tumor and adjacent non-tumor expressions projected onto a most discriminative orientation for differentiating between different patients . As to each feature pair, expressions from cancer and adjacent normal tissues were viewed as a matrix form. That is, the expressions of each sample on each feature pair formed a second-order matrix, which is composed of two column vectors derived from the the cancer and normal tissue of each feature. Accordingly, a bilinear form and matrix-variate discriminative analysis30 was emulated in order to overcome the problem of small sample size compared with large feature numbers. As illustrated in Fig. On assumption that expression profiles are composed of three dimensions including feature , sample  and tissue , joint covariate detection embodies not only a bilinear projection on tissue and feature dimension, but also IHT on sample dimension. Besides, an A5 formulation21. Among them, 29 samples were associated with metastasis cases. The other 102 samples corresponded to non-metastasis cases. Missing values of the downloaded normalized data were imputed using k nearest-neighboring algorithm in Euclidean distance. Probes associated with human miRNAs were extracted for further analysis. We utilized the A5-based feature selection method for searching individuals and pairs of probes expressed differently between metastatic and metastatic-free HCC group.In order to discover metastasis-related miRNAs differentially expressed in HCC, 131 primary HCC patients with metastasis or non-metastasis cases were confirmed from a public dataset with its accession number GSE685726, which were displayed in detail from Equations (41) to (43) in that paper. Here, we presented an approximate implementation. This simplification converted the bilinear form to two separated learning steps on projection directions of not only the HCC accompanied with its adjacent normal tissues but also each pair of probes. In other words, the bilinear form corresponded to a two-step Fisher’s linear projection, i.e., combined projection and dimension reduction projection, as shown in Fig. Above all, a combination needed to be made between HCC and adjacent normal expression profiles of each individual probe for a better discriminative performance. Therefore, Fisher’s LDA which kept the smallest variance within each phenotype group  and provided a most discrimination between two groups was utilized. As to each probe pair, expressions of HCC and adjacent normal tissues formed a three-dimensional matrix, of which the dimensions were along the probe pair, two phenotype groups of patients  and HCC accompanied with its adjacent normal tissues. Xu proposed a bilinear form and matrix-variate discriminative analysis of microarrays26. Model-based perspective was equivalent to common hypothesis testing that developed a statistics to evaluate how different it could be between two populations of samples. As to boundary-based perspective, classifying each sample to the right phenotype group it belonged to could evaluate the performance using classification error rates by a formed hyperplane. In order to disclose the joint performances of IHT, we displayed a complementary nature of IHT. To each individual and each pair of probes, we made a permutation test based on Welch’s t-test after combined projection at step A1 (see Methods). P-values corresponding to each probe were obtained by 1 × 104 random permutations of the class label . As to each pair of probes, a stepwise strategy was made in view of the larger amount of computation time for permutations on enumeration of each pair. We performed 1 × 104 random permutations of the class label. Pairs with the smallest p-values (p = 0.0001) were selected for further 1 × 105 random permutations. This procedure was repeated until the permutation times were 1 × 106. Besides, Fisher’s LDA was also utilized as a classifier to each individual and each pair of probes (see Methods). Combined projection itself at step A1 formed a classifier on each probe. As to each pair, dimension reduction projection was provided. 1 × 104 random re-sampling was made for cross-validation with 70% of the two groups  selected as a training set each time and the left samples as a testing set. The classification error rate was defined as an arithmetic mean between the two phenotype groups and calculated each time, considering that the sample size was not balanced between metastasis and metastasis-free group. As a result, an average of classification error rates from 1 × 104 testing sets was obtained. Guided with the IHT thought, the average of classification error rates on testing sets and the corresponding p-value from the same individual and pair of probes formed a 2D scattering point. Together, 2D scattering points from all the enumerations formed a 2D scattering plot, as shown in Fig. As has been declared, IHT was composed of not only model-based perspective but also boundary-based perspectiveFrom Fig. Anyway, interception of individuals or pairs with the top 10% p-values was too subjective. Thus, we considered to perform a bootstrap technique by selecting 90% samples in each round. We kept calculating the p-value and the average of classification error rates of each individual and pair of probes in each round. Then, we ranked individuals and pairs of probes by p-values and averages of classification error rates in an ascending order, respectively. Using the two orders, we voted for each individuals and pairs with a strategy as follows. Individuals or pairs with their rankings from No. 1 to No. 3 kept 20 scores. Those at the ranking from No. 4 to No. 5 obtained 15 scores. Those with their ranking from No. 6 to No. 10 kept ten scores. Those with their positions from No. 11 to No. 15 got five scores. Those gained one score with their rankings from No. 16 to No. 20. This strategy kept summing the scores from Welch’s p-value and the average of classification error rates of each round, when both of the two rankings were at the first 20. Otherwise, no score would be accumulated in this round. After 100 rounds of cycling step A2 and step A3, we obtained the accumulated scores of combinational rankings listed in Supplementary Tables At step A4, we chose those with their overall scores bigger than 200 for further analysis according to the strategy of assigning scores at step A3. In other words, individual or pair of probes gained at least two scores on average in each round should be chosen for further analysis. As a result, we selected 15 individuals of probes and 27 pairs of probes for further affirmation  that indicated the relationships between group II and group IV see Fig. . Besides35 exist for selecting genes on expression profiles. In order to illustrate the effectiveness of our method, we selected recursive cluster elimination (RCE)33 and random forest (RF)35 for feature selection on dataset GSE6857 and made comparisons with our method. Following the steps of algorithm SVM_RCE 33, we obtained 18 miRNA probes. After 1 × 104 random re-sampling for cross-validation with 70% of the two groups  selected as a training set each time and the left samples as a testing set on combined projection, we got the average accuracies using SVM and Fisher’s LDA. Meanwhile, we made a permutation test at 1 × 104 times based on Welch’s t-test after combined projection. The corresponding p-value was calculated. As to RF35, same evaluating indicators were calculated on 30 miRNA probes derived from RF, except that we changed the average accuracy of SVM to that of RF. As shown in Fig. Many algorithms36, GSE67138 and GSE67139) for further independent validation. GSE76903 kept 20 patients with primary tumor, portal vein tumor thrombosis (PVTT) and adjacent normal. As to GSE67138 and GSE67139, 57 and 120 different patients with either only primary tumor or tumor vascular invasion were considered. Thus, we made a bilinear projection on GSE76903 and a dimension reduction projection on GSE76903, GSE67138 and GSE67139 using the selected pairs of miRNAs to be significant. After 1 × 104 random re-sampling for cross-validation with 70% of the two groups  selected as a training set each time and the left samples as a testing set, we obtained the average accuracies using Fisher’s LDA. Besides, p-values corresponding to a permutation test at 1 × 104 times based on Welch’s t-test were calculated. The experimental results were listed in Table In order to shown the effectiveness the selected pairs of miRNAs considered to be signaficant, we chose three datasets  associated with venous metastasis in primary HCC. Among them, miR-210First, we practically combined tumor and adjacent non-tumor expressions together based on Fisher’s LDA. Inevitably, useful information was discarded when tumor and adjacent non-tumor tissues were treated separately. In addition, adjacent non-tumor tissues were always regarded as background, and that led to an inappropriate subtraction between the tumor log2 expressions and the corresponding background. In fact, this only provided a linear combination using a special linear projection on the counter-diagonal orientation with tumor and adjacent normal expressions to be coordinates. Instead, Fisher’s LDA-based combination was utilized, which corresponded to an optimal combination with a best discriminative performance.20. However, it was still a linear projection between the expressions from two different miRNAs on diagonal or counter-diagonal orientation. In order to solve the problems mentioned above, a matrix-variate hypothesis testing26 that considered tumor and adjacent non-tumor expressions from two different miRNAs was introduced in. Essentially, we simplified bilinear projection as a two-step linear projection  perform perfect. Actually, this phenomenon derives from the existence of probably potential metastasis in metastasis-free group on dataset GSE6857. Anyway, the average accuracy of our method using Fisher’s LDA is better than that of RCE and RF, especially that the performance of the selected miRNA pairs considered to be significant using our enumeration method is better than that of the miRNA groups derived from RCE and RF. This phenomenon further emphasize that the components of each significant miRNA pair may derive from different miRNA cliques. As shown in Table From Table 37 was thought to participate in migration of HCC, although it was dispensable for p16-enhanced migration. CCND138 was regarded to account for cell proliferation, recurrence and metastasis in HCC. Besides, functional association of MDM239 binding protein with metastatic potential of hepatocellular carcinoma was found. All these literatures indirectly supported that parts of p53 signaling pathway  participate in HCC venous metastasis, which might provide the evidences of identified miRNA pairs and further certificate the effectiveness of our method.Significantly different from pathways associated with each component of a significant miRNA pair, two pathways in bold were achieved in Tables w with the most discriminative performance was expressed as,m1 and m2 were the 2-dimensional sample mean of HCC and adjacent non-tumor tissues for combined projection or the 2-dimensional sample mean of column vectors representing combined projection values on a miRNA pair for dimension reduction projection, respectively. Sw was called the within-class scatter matrix, and was given by,x denoted a 2-dimensional sample from HCC or adjacent non-tumor tissues for combined projection or a 2-dimensional sample from combined projection values for dimension reduction projection. n1 and n2 were the numbers of samples that represented metastasis-free and metastasis samples with n1 = 102 and n2 = 29. Detail formula derivation could be seen in Duda’s book25. The Fisher’s LDA-based classification was based on the optimal decision boundary with its equation wtx + w0 = 0, where w0 was given by x was regarded as a non-metastasis sample when wtx + w0 < 0, and vice versa. Therefore, the error rate that wrongly classified x into the non-metastasis group was defined as x into the metastasis group was Er = (Er1 + Er2)/2.We utilized Fisher’s LDA to fulfill combined projection between HCC and adjacent normal tissues and accomplish dimension reduction projection within each miRNA pair. The orientation from Fisher’s linear projection was used to separate the projected samples. Weight vector m1(i), m2(i) and i-th probe or probe pair from enumeration. The i-th freedom degree was defined as i-th t statistics obeyed the same distribution regardless of how we made the assignments of group labels. Therefore, the p-value for the i-th statistic was calculated by,t0 represented a null statistics by a random rearrangement of class label with B to be the times of permutation.We tested combined projection values of each individual of probes or dimension reduction projection values of each pair of probes using Welch’s t-test, which was expressed as,Er based on boundary perspective with p-value derived from model perspective. Considering different scales about the spatial distribution of scatters  representing individuals or pairs of probes, a voting strategy was made. In other words, individuals or pairs of probes were ranked by p-values and averages of Ers in an ascending order, respectively. Votes from the two aspects were accumulated together in order to select candidate individual miRNAs or pairs. In detail, the voting strategy was proposed as Table In each round of re-sampling, IHT combined the error rate Supplementary materials