We apply it to the benchmark of single-cell RNA-sequencing analysis pipelines using simulated and real datasets with known cell identities, covering common methods of filtering, doublet detection, normalization, feature selection, denoising, dimensionality reduction, and clustering. pipeComp can easily integrate any other step, tool, or evaluation metric, allowing extensible benchmarks and easy applications to other fields, as we demonstrate through a study of the impact of removal of unwanted variation on differential expression analysis.We present  Single-cell RNA-sequencing (scRNAseq) and the set of attached analysis methods are evolving fast, with more than 560 software tools available to the community , roughlyA number of good comparison and benchmark studies have already been performed on various steps related to scRNAseq processing and analysis and can guide the choice of methodology –21. Howemuscat [Seurat pipeline as the starting point to perform an integrated investigation of (1) doublet identification, (2) cell filtering, (3) normalization, (4) feature selection, (5) dimension reduction, and (6) clustering. We compared competing approaches and also explored more fine-grained parameters and variations on common methods. Importantly, the success of methods at a certain analytical step might be dependent on choices at other steps. Therefore, instead of evaluating each step in isolation, we developed a general framework for evaluating nested variations on a pipeline and suggest a multilevel panel of metrics. Finally, we evaluate several recent methods and provide practical recommendations.Here, we develop a flexible R framework for pipeline comparison and apply it to the evaluation of the various steps of analysis leading from an initial count matrix to a cluster assignment, which are critical in a wide range of applications. We collected real datasets of known cell composition Table  and usedmuscat . Since gmuscat  was prevmuscat , 15, we pipeComp package defines a pipeline as, minimally, a list of functions executed consecutively on the output of the previous one , are fairly frequent in scRNAseq datasets, with estimates ranging from 1 to 10% depending on the platform and cell concentration used , 27. WhiDoubletFinder [scran’s doubletCells [scds [DoubletFinder integrates a thresholding based on the proportion of expected doublets, while scran and scds return scores that must be manually thresholded. In these cases, we ensured that the right number of cells would be called doublets. In addition to these methods, we reasoned that an approach such as DoubletFinder could be simplified by being applied directly on counts and by using a pre-clustering to create neotypic/heterotypic doublets more efficiently. We therefore also evaluated a simple and fast Bioconductor package implementing this method for doublet detection, scDblFinder, with the added advantage of accounting for uncertainty in the expected doublet rate and using meta-cells from the clusters to even include triplets  and even in a simulation (simMix1) that should not contain any , the other two datasets proved more difficult Fig. a. scDblFhod Fig. b. Acrosshod Fig. c. We alsscater that measures, for a given control property, the number of median absolute deviations (MADs) of each cell from the median of all cells. Additional File Beyond doublets, a dataset might include low-quality cells whose elimination would reduce noise. This has for instance been demonstrated for droplets containing a high content of mitochondrial reads, often as a result of cell degradation and resulting loss of cytoplasmic mRNAs . A commoReasoning that the cells we wish to exclude are the cells that would be misclassified, we measured the rate of misclassification of each cell in each dataset across a variety of clustering pipelines, correcting for the median misclassification rate of the subpopulation and then evaluated what properties could be predictive of misclassification  and two methods based on scater’s runPCA using all  or selected covariates (pca.sel). An important risk of excluding cells on the basis of their distance from the whole distribution of cells on some properties  is that these properties tend to have different distributions across subpopulations. As a result, thresholds in terms of number of MADs from the whole distribution can lead to strong biases against certain subpopulations . Seurat’s scaling function (and sctransform) also includes the option to regress out the effect of certain covariates. We tested its performance by using the proportion of mitochondrial counts and the number of detected features as covariates. Finally, it has been proposed that the use of stable genes, in particular cytosolic ribosomal genes, can be used to normalize scRNAseq [We next investigated the impact of different normalization strategies. Besides the standard log-normalization included in lization , sctransormation , normaliormation , 36, and [SCnorm  , but very poorly on others, in terms of both the subpopulation silhouette and clustering accuracy . Scaling ” vs “seurat”) tended to reduce the average silhouette width of some subpopulations and to increase that of some less distinguishable ones and was generally, but not always, beneficial on the accuracy of the final clustering. Regressing out covariates systematically gave equal or poorer performance on MI, ARI, and ARI at true number of clusters, with a strongest negative impact among the Smart-seq datasets (Koh and Kumar). Regressing out covariates using sctransform instead tended to have a much milder effect, possibly owing to its regularization. In general, sctransform systematically outperformed other methods and, even though it was developed to be applied to data with unique molecular identifiers (UMI), it also performed fairly well with the Smart-seq protocol (Koh and Kumar datasets).We further evaluated normalization methods by investigating their impact on the separability of the subpopulations. Since clustering accuracy metrics such as the ARI are very strongly influenced by the number of clusters, we complemented it with silhouette width  and mutusctransform) or lower  number of clusters, the effect was very mild and not entirely systematic  rely on a different measure of variance, again standardized. While adjusting for the mean-variance relationship removes much of the bias towards highly expressed genes, it is plausible that this relationship may in fact sometimes reflects biological relevance and would be helpful in classifying cell types. Another common practice in feature selection is to use those with the highest mean expression. Recently, it was instead suggested to use deviance [sctransform provides its own ordering of genes based on transformed variance.A standard clustering pipeline typically involves a step of highly variable genes selection, which is complicated by the digital nature and the mean-variance relationship of (sc)RNAseq. deviance , while ssctransform normalization, and deviance explained. The ANOVAs performed on a standard Seurat normalization and on sctransform data were highly correlated  but did not perform so well to select genes with a high deviance explained.Reasoning that a selection method should ideally select genes whose variability is higher between subpopulations than within, we first assessed to what extent each method selected genes with a high proportion of variance or deviance explained by the  subpopulations. As the proportion of variability in a gene attributable to subpopulations can be measured in various ways, we first compared three approaches: ANOVA on log-normalized count, ANOVA on We next evaluated how the use of different feature selection methods affected the clustering accuracy Fig. b. To valSeurat’s PCA, scran’s denoisePCA, and GLM-PCA [sctransform normalization. Given that Seurat’s default PCA weights the cell embeddings by the variance of each component, we also evaluated the impact of this weighting with each method.Since the various PCA approaches and implementations were recently benchmarked in a similar context , we focu GLM-PCA . When reSeurat’s PCA procedure however proved superior on all metrics. Like GLM-PCA, scVI’s Linearly Decoded (LD) data [Seurat’s PCA in view of the silhouette width  and harboring many subpopulations (simMix1 and Zhengmix8eq).The impact of the dimensionality reduction methods tested was far greater than that of normalization or feature selection  data  and lateSeurat’s resolution parameter had a much stronger impact.A common step following dimension reduction is the selection of an appropriate number of dimensions to use for downstream analysis. Since Euclidean distance decreases as the number of non-discriminating dimensions increases, there is usually a trade-off between selecting enough dimensions to keep most information and excluding smaller dimensions that may represent technical noise or other unwanted sources of variation. Overall, increasing the number of dimensions robustly led to a decrease in the number of clusters , to more complex algorithms. We evaluated the performance of dimensionality estimators implemented in the  package , as well package  or scrannoisePCA , and thenoisePCA . We alsonoisePCA , but whinoisePCA .sctransform or standard Seurat normalization.We compared the various estimates in their ability to retrieve the intrinsic number of dimensions in a dataset, based on Seurat’s weighted PCA space. As a first approximation of the true dimensionality, we computed the variance in each principal component (based on Seurat’s weighted PCA space) that was explained by the subpopulations, which sharply decreased after the first few components in most datasets Fig. a. These MCVR and the global maximum likelihood based on translated Poisson mixture model  tended to produce clusterings with smaller deviations to the true number of subpopulations , which violates the assumptions of the method, and while it did increase the separability of some subpopulations  between the number of dimensions called by MCVR and the log10 average library size of a dataset, which explained the former considerably better than the number of subpopulations.ons Fig. c, it wasmaxLikGlobal) systematically selected many more components than were associated with the subpopulations in Fig. Of note, the optimal method in our evaluation  or the optimization of the modularity score (fast_greedy) on various underlying graphs. Specifically, we compared rank-based graphs (scran’s default) with graphs using the components’ values directly, SNN vs KNN graphs, and the exact nearest neighbors to the Annoy approximation. Again, the tested methods were combined with Seurat’s standard normalization and sctransform, otherwise using the parameters found optimal in the previous steps.The last step evaluated in our pipeline was clustering. Given previous work on the topic , 7 and tSince ARI is dominated by differences in the number of clusters  rather than committing misclassification errors. Overall, the walktrap method appeared superior to the fast greedy algorithm and was generally comparable to Seurat clustering, although the latter offered more control over the resolution. A major difference between walktrap-based clustering and Seurat is the computing time  did improve the separability of some more elusive subpopulations, no method had a systematically positive impact on the average silhouette width across all subpopulations  consistently underperformed. 10X datasets, which are typically characterized by a lower per-cell coverage and feature detection rate, benefited more from imputation and, in this context, DrImpute and DCA tended to show the best performance. On the contrary, imputing on normalized counts originating from Smart-seq technology was instead rather deleterious to the clustering accuracy.The basic pipeline presented here can be extended with additional analysis steps while keeping the same evaluation metrics. To demonstrate this, we evaluated various imputation or denoising techniques based on their impact on classification. Since preliminary analysis showed that all methods performed equally well or better on normalized data, we applied them after filtering and normalization, but before scaling and reduction. Although some of the methods -based methods on bulk RNAseq differential expression analysis (DEA). The rationale behind such approaches is that the identification of factors determining genome-wide variation  and their inclusion in the DEA’s generalized linear model might improve DEA, especially in the presence of sources of variation unrelated to the experimental variables. sva [RUVseq [seqc and simulation), and benefited all DEA methods in a similar fashion , DrImpute (with prior processing), and DCA to offer the best performances, although none of these methods was beneficial over all subpopulations.We found especially Denoising/imputationSeurat.Similarly to previous reports , we recoglobal maximum likelihood based on translated Poisson mixture model method .We advise against the commonly used elbow method (because it is too conservative) and the jackstraw method  for deciding on the components to include and recommend a method like the PCASeurat can be useful in affording manual control of it while, in the absence of such knowledge, scran-based walktrap clustering provided reasonable estimates, although at greater computational costs.In cases where prior knowledge can guide the choice of a resolution, ClusteringOn the basis of our findings, we can make a number of practical analysis recommendations, also summarized in Additional File In this study, we evaluated tools commonly used for the processing of scRNAseq with a focus on droplet-based datasets, namely from the 10x technology. Although this platform has been used in almost half of the scRNAseq studies in 2019 –56. AlthAmong all tested pipelines, performance varied greatly across datasets. The ZhengMix8eq and simMix1 datasets were the most challenging datasets across all preprocessing steps that we tested. The average silhouette width of half of their subpopulations was systematically lower than the other subpopulations/datasets and the ARI score at true number of clusters, if ever achieved, was always poor. It is likely that these two datasets are challenging due to their combined sparsity and complexity , of different datasets  and the use of different filtering and normalization strategies. ALRA however yielded good results in both studies and may be the most conservative option. The study from Hou et al. also highlighted the usefulness of ALRA when followed by differential and trajectory analysis both in UMI and Fluidigm technologies.Our recommendations are in line with the results of several other benchmarking studies. For instance, formance  and DrImanalyses . A previanalyses  instead u et al.  that recscVI in our evaluation was lower than in the original publication presenting the method [scVI’s latent space. These studies, like ours, used datasets with relatively few cells , for which scVI was reported by its authors to be less adapted. We think that further work will be required to assess its performance on larger datasets.The performance of e method . Our rese method  and cluse method  of scVI’pipeComp framework is modular and flexible enough to integrate new steps in the pipeline, as shown by an example with imputation/denoising methods.Here, we chose to concentrate on what could be learned from datasets with known cell labels . In conpipeComp to address the need of a framework that can simultaneously evaluate the interaction of multiple tools. The work of Vieth and colleagues[CellBench package was recently proposed to address a similar need [pipeComp is its ability to perform evaluation in parallel to the pipeline and offering on-the-fly evaluation at multiple steps. This avoids the need to store potentially large intermediate data for all possible permutations and thus allowing a combinatorial benchmark. This, along with the set-up simplicity (demonstrated here by our separate SVA-DEA benchmark), also distinguishes pipeComp from general workflow frameworks such as drake [CellBench and drake). In addition, the fact that benchmark functions are stored in the PipelineDefinition makes the benchmark more smoothly portable, making it easy to modify, extend with further methods, or apply to other datasets.We developed olleagues already lar need , offerinas drake ; indeed,DrImpute is in line with a previous study focused on the preservation of data structure in trajectory inference from scRNAseq [ALRA was however not included in this previous benchmark). Our results also align with the hypothesis of this study on the respective strengths of linear and non-linear models and their use to different types of data, such as the highest performance of non-linear methods in developmental studies.With respect to the methods themselves, we believe there is still space for improvement in some of the steps. Concerning cell filtering, we noticed that the current approach based on whole-population characteristic  can be biased against certain subpopulations, suggesting that more refined methods expecting multimodal distributions should be used rather than relying on whole-population characteristics. In addition, most common filtering approaches do not harness the relationship between cell QC properties. Finally, imputation had a varied impact on the clustering analysis and seemed to be linked to the technology that was used to generate the data. Also, the good performance of scRNAseq   website. Both RNA and ADT count data were downloaded from GEO and processed independently using g muscat  and mergg muscat  in a simscDblFinder was used for doublet identificationThe default filter sets (see below) were appliedSeurat normalization was employedStandard Seurat (≥3.0) variable feature selection was employed, selecting 2000 genesSeurat scaling and PCA were employedStandard Seurat clustering analyses were performed, selecting different numbers of dimensions  and using various resolution parameters . The range of resolution parameters was selected to ensure that the right number of clusters could be obtained in all datasets.To study the impact of upstream steps on clustering, various Where unspecified, the following default parameters or parameter sets were used:DrImpute documentation advises to process the data prior to imputation by removing lowly expressed genes and cells expressing less than 2 genes. As it is not clear if this step is a hard requirement for the method to perform well, DrImpute was run with and without prior processing . The DCA method only accepts integers counts, while two of the datasets had non-integer quantification of expected counts. For these datasets, we rounded up the counts prior to imputation. ALRA is designed for normalized data but as we are evaluating normalization downstream to imputation, we used the method on both non-normalized  and normalized counts . scImpute requires an estimation of the expected number of clusters with the input data. As the estimation of the true number of cluster may not be known by the user, we evaluated the tool using the true number of clusters (scImpute label) and using an over/under-estimation of this number . ENHANCE uses a k-nearest neighbor aggregation method and automatically estimates the number of neighbors to merge prior to the imputation. With the smallest datasets, this parameters was estimated to be 1, which lead to an early stop of the function. In such cases, we manually set this parameter to 2 for the method to work.Most imputation/denoising methods were run with the default parameters with the following exceptions; Seurat documentation advises not to use a p value threshold  but rather look for a drop in significance, we applied the same farthest point algorithm on the log10, which in our hands reproduced manual threshold selection.For the methods that produce local dimensionality estimates, we used the maximum. For the elbow method, we implemented an automatic procedure by taking the farthest point from a line drawn between the variance explained by the first and last  components calculated. For the JackStraw method, since the https://github.com/constantAmateur/MCVR), with minor modifications (see wrapper in the pipeComp package). Briefly, since there is no a priori relationship between the dimensionality of the count data and that of the PCA based on the normalized data, MCVR does not use the original approach based Poisson loss, but the mean squared error of the normalized data, and uses balanced 50-50 splits. Because of the 50% split, dimensionality may be underestimated, and titration did suggest that was the case for some of our datasets.For the molecular cross-validation approach, we relied on a R implementation by Matthew Young  > or <5 MADs, pct_counts_Mt >2.5 MADs and >0.08.The stringent set of filters uses the same thresholds, but excludes a cell if it is an outlier on any single distribution. The veryStringent set of filters excludes any cell that is an outlier by at least 2 MADs (lower or higher) on any of the following: log10_total_counts, log10_total_features, pct_counts_Mt, or pct_counts_in_top_20_features. The lenient set of filters excludes cells that are outliers on at least two distributions by at least 5 MADs, except for pct_counts_Mt where the threshold is >3 MADs and >0.08.The scran::quickCluster and the filters were then applied separately for each cluster.For cluster-wise filters, clusters were first identified with scater, running runPCA with use_coldata=TRUE, detect_outliers=TRUE. “pca.all” uses all covariates, while “pca.sel” uses only the log10(counts), log10(features), proportion mitochondrial, and proportion in the top 50 features.The “pca.all” and “pca.sel” clusters refer to the multivariate outlier detection methods implemented in sctransform. For each gene, we fitted a linear model using the subpopulation as only independent variable (∼subpopulation) and used the R-squared as a measure of the variance explained. The same method was used for principal components.Unless specified otherwise, the variance in gene expression explained by subpopulations was calculated on the data normalized and transformed through pipeComp. The function uses edgeR to fit two models, a full (∼librarySize+population) and a reduced one (∼librarySize). For each gene, the deviance explained is then the difference between the deviance of each models, divided by the deviance of the reduced model. In the rare cases where this resulted in a negative deviance explained, it was set to zero.The deviance explained by subpopulations was calculated directly on counts using the getDevianceExplained function of To estimate the correlation between the principal components and covariates such as library size, we first fitted a linear model on the subpopulations and correlated the residuals of this model with the covariate of interest.scVI algorithm was performed separate to the other pipeline combinations since the model training on our reference datasets was especially resource-consuming and follows a different workflow from the other pipelines. In particular, for datasets with fewer cells than genes, scVI requires a feature selection to be performed prior to the normalization step; furthermore, its dimensional reduction steps rely directly on raw counts. We therefore evaluated it separately and compared it to a representative subset of pipelines using sctransform normalization and Seurat’s PCA dimension reduction method. The resulting evaluation is available in the Additional File The evaluation of scVI, we considered as normalized values the normalized mean of the negative binomial distribution modeling the counts . Following scVI’s online documentation, we used the VAE model, using a train size of 1, a training on 400 epochs, and a learning rate of 0.001. As a major limitation of scVI is the training on dataset with low gene to cell ratio, the features were subsampled to 2000 using the subsample_genes function, which resembles the Seurat V3 VST gene selection method. All other parameters were left unmodified.Using scVI package computing a dimension reduction on the data were tested. First, we evaluated the mean of the posterior distribution for the latent space, still based on the VAE model (accessed through the get_latent function). All other parameters were set as described above or left as default. The second model that we evaluated was the linearly decoded VAE of scVI (LDVAE model). It was trained on 250 epochs, a learning rate of 0.001 and computed 50 dimensions. For some smaller datasets, the training phase aborted. In such cases, we followed the recommendations of the package and reduced the learning rate by a factor 10 until the model could properly converge. Given that the other methods that we evaluated returned PCs ordered by their variance explained, we applied to the same reordering to the latent space returned by scVI. For both models. we relied on the feature selections results from the pipeline and not on the subsample_genes function. All other parameters were left unmodified.Two models of the The set of alternatives found for the overall run  from the iPSCpower package [For the  package  and seleseqc dataset, we used data from the seqc package [For the  package . For botsimulation, we simulated counts using the polyester package [For the  package  and the The three benchmark datasets are available in vstsva, we first applied the variance-stabilizing transformation from DESeq2 [edgeR.QLF refers to the quasi-likelihood version [edgeR’s fit. The exact wrappers used can be found in the package at https://github.com/plger/pipeComp/blob/master/inst/extdata/dea_wrappers.R.Filtering was performed with edgeR’s filterByExpr function, changing the min.count argument. For m DESeq2  before a version  of edgeRSeurat (version 3.0.0), sctransform (0.2.0), DoubletFinder (2.0.1), scDblFinder (1.1.4), scds (1.0.0), SAVERX (1.0.0), scImpute (0.0.9), ALRA (commit 7636de8), DCA (0.2.2), DrImpute (1.2), ENHANCE , and SAVERX (1.0.0). The glmPCA was performed using the glmpca package, while the code for deviance calculation was obtained from https://github.com/willtownes/scrna2019 (commit 1ddcc30ebb95d083a685f12fe81d35dd1b0cb1b2).Analyses were performed in R 3.6.0 (Bioconductor 3.9) and the following packages were installed from GitHub repositories: Additional file 1 Supplementary Figures (PDF 16 MB)Additional file 2 More detailed description of pipeComp and comparison with other tools. (PDF 0.5 MB)Additional file 3 Review history.


Some carcinomas show that one or more metastatic sites appear with unknown origins. The identification of primary or metastatic tumor tissues is crucial for physicians to develop precise treatment plans for patients. With unknown primary origin sites, it is challenging to design specific plans for patients. Usually, those patients receive broad-spectrum chemotherapy, while still having poor prognosis though. Machine learning has been widely used and already achieved significant advantages in clinical practices. In this study, we classify and predict a large number of tumor samples with uncertain origins by applying the random forest and Naive Bayesian algorithms. We use the precision, recall, and other measurements to evaluate the performance of our approach. The results have showed that the prediction accuracy of this method was 90.4 for 7,713 samples. The accuracy was 80% for 20 metastatic tumors samples. In addition, the 10-fold cross-validation is used to evaluate the accuracy of classification, which reaches 91%. Tumors can develop in any part of body, and some tumors even can metastasize to other parts of the body from their primary sites after developing at a certain point. In general, the occurrence of tumors at primary sites and their metastatic sites could be found deferentially, and the primary origins of metastatic cancers can be identified within a short amount of time by clinical assessments . HistoloThe targeted therapy  can be uTOO is a product of array-based gene expression profiles. TOO can identify 2,000 genes and 15 types of tumors, including thyroid cancer, breast cancer, non-small cell lung cancer, pancreatic cancer, gastric cancer, colorectal cancer, liver cancer, bladder cancer, kidney cancer, non-Hodgkin’s lymphoma, melanoma, ovarian cancer, sarcoma, testicular germ cell tumor, and prostate cancer. The advantage of this product is that it prevents the subjective bias. It can objectively identify the primary origin of cancers no matter which is well-differentiated or not. However, TOO is time-consuming, which is not feasible for clinical practices .CancerTYPE ID is a product that uses cancer samples based on RT-PCR data. In the study , 578 labThough the above two products have good performance for some types of cancers, two products are costly with up to $3000–$4000 , and theFirstly, data were downloaded from TCGA and GEO. Secondly, after data preprocessing for raw data, genes were selected by the random forest algorithm with 10-fold cross-validation. Finally, the naive Bayes classifier was used to classify the 20 kinds of tumors, and the output of the model was shown as the evaluation index. The detailed step is shown in A total of 7,715 RNA-seq samples that covered 21 cancers and excluded metastatic cancers were extracted from TCGA. In the process of data preparation, we eliminated two samples due to the lack of clinical data. Therefore, the remaining 7,713 samples were used as either the training dataset or the validation dataset for the classification. Furthermore, the expression spectrum matrix of 7,633 samples was constructed. Each sample contained 20,501 genes. In this paper, 372 samples from metastatic cancers were selected as the test dataset, of which 352 samples belong to Skin Cutaneous Melanoma (SKCM). The ratio of SKCM was much higher than other types of metastatic cancers, and we excluded SKCM data from our selected data in order to reduce the possible effects on the results. The detailed information of selected data is shown in For the independent validation dataset, 48 samples are obtained from GEO and processed according to the description in section “Materials and Methods” and then used for the trained naive Bayesian model to make the prediction. The detailed information of selected data is shown in ∗N was formed, with M representing the number of samples and N representing the numbers of genes, and all samples were labeled with the type of each cancer. The expression profile was divided into 20 types of cancer, and the combination of five genes could be used to classify this problem , decision tree, and Naive Bayesian to classify 20 cancers is shown in The comparison among results of the 1. Each sample and each gene from each cancer type table were extracted to generate a matrix of M∗N, where M is the number of samples, N is the number of genes, and all the samples were labeled by cancer types. All primary tumors were divided into training sets and metastatic tumors were divided into test sets.The TCGA RNA-seq and array data were downloaded from the ICGC Data PortalFor the independent set, 48 samples from 5 known cancer origin sites were downloaded from Gene Expression Omnibus (GEO). These tumors belong to GSE10907, GSE11222, GSE5608, GSE8352, GSE4895, GSE8912, GSE7966, and GSE12281. In addition, these 5 cancers belong to the 20 cancer types in this paper.1,g2,.,gn} stands for feature, and C stands for cancer type. That is, to calculate the Gini score Sj for each feature gj, the calculation formula of Gini index is as follows:In order to reduce the cost of gene number determined by gene panel, in this study, random forest algorithm was applied to select genes. The Gini average impurity in random forest was used as the criterion to estimate the importance of genes. The random forest is composed of several decision trees, which are binary decision trees. Each node in the decision tree is a condition on a single gene. As a result, we can achieve the goal by splitting the dataset into two datasets; therefore, a similar expression level can be classified in the same dataset. For random forest, the average reduction of each feature impurity can be calculated. In addition, the importance score of genes can be calculated and sorted according to the score. GI stands for Gini, S stands for importance score, G = {gmk represents the proportion of category k in node m.where c represents C categories, and Pj in node m, that is, the variation of Gini impurities before and after node m branch, is calculated as follows:The importance of feature gl and GIr, respectively, represent the Gini index of the two new nodes after branching, and Sjm represents the importance of feature gj in node m.where GIj that appears in decision tree i belongs to M, the importance of gj in the ith tree is calculated as follows:If the node m with characteristic gAssuming the random forest has t trees, the importance score formula of forest is:The importance score is obtained by normalizing all the importance scores obtained:The top N genes with high scores were selected until the stopping criterion was met. Finally, the selected genes in all samples participated in the next classification.Using the gene ontology  as the dIn this paper, naive Bayes was used as the classifier of gene combination. Naive Bayes is one of the classical machine learning algorithms. It is a classification algorithm based on Bayes theorem. Its principle is simple and easy to implement. The core idea of naive Bayesian algorithm is to assume that each feature is independent. For a given type of data to be judged, classify and predict according to the training dataset, and calculate the probability that the current type of data to be judged belongs to a certain category through Bayesian theorem. The maximum probability relationship obtained is that the algorithm judges the category of these data. Naive Bayesian algorithm can be divided into three parts:k represents the kth category, gi represents the ith feature, and then calculate conditional probability by prior probability. The formula is as follows:First, determine the feature attributes; that is to say, the expression profiles of 2,284 genes corresponding to each sample were extracted. Then, it was assumed that all the features conformed to the Gauss distribution. The samples in the dataset were labeled as cancer type. G represents the characteristics and C represents the type of cancer, which can be calculated as the prior probability P(C). CThe conditional probability of all the kth classes is calculated by the Bayesian formula:Since all the features conform to the Gaussian distribution and are independent of each other, the formula for conditional probability becomes as follows:i is the ith feature, and μi,ck and σi,ck are the mean and variance of the ith feature in the K class Ck, respectively.where gThe conditional probability formulas for all the Kth class are calculated as follows:k| G), that is:Finally, obtain the relationship between the maximum probability data to be classified and the category, P Co., Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


EML4-ALK). Even though RNA sequencing (RNA-Seq) technologies combined with existing bioinformatics approaches have enabled researchers to systematically identify fusion transcripts, specifically detecting f-circRNAs in cells remains challenging owing to their general sparsity and low abundance in cancer cells but also owing to imperfect computational methods.In cancer cells, fusion genes can produce linear and chimeric fusion-circular RNAs (f-circRNAs), which are functional in gene expression regulation and implicated in malignant transformation, cancer progression, and therapeutic resistance. For specific cancers, proteins encoded by fusion transcripts have been identified as innovative therapeutic targets  actual synthetic spike-in RNA-Seq data, (ii) simulated RNA-Seq data, and (iii) actual cancer cell–derived RNA-Seq data. Fcirc showed significant advantages over existing methods regarding both detection accuracy  and computing performance .Fcirc is a powerful and comprehensive Python-based pipeline to identify linear and circular RNA transcripts from known fusion events in RNA-Seq datasets with higher accuracy and shorter computing times compared with previously published algorithms. Fcirc empowers the research community to study the biology of fusion RNAs in cancer more effectively. EML4-ALK rearrangements [Various events such as gene mutations, gene rearrangements, and chromosomal fragile sites are able to induce the formation of fusion genes in the genome of cancer cells . These fngements . Therefongements  and circngements , the curngements , and curngements .Therefore, in the present study, we developed “Fcirc,” a comprehensive, accurate, and free-of-charge pipeline to analyse RNA-Seq data for linear and circular RNAs transcribed from fusion genes.EWSR1-ATF1, TMPRSS2-ETV1, EWSR1-FLI1, NTRK3-ETV6, CD74-ROS1, HOOK3-RET, EML4-ALK, AKAP9-BRAF, and BRD4-NUTM1. The sequencing data (Illumina HiSeq 2500 system) were made available in FASTQ format in the SRA under accession number SRP043081 and allow researchers to validate novel algorithms for gene fusion detection in a comparative manner.To evaluate the performance of different tools for fusion RNA analysis, we took advantage of RNA-Seq data from a study performed by Tembe et al. . In thisThe simulator art_illumina function in ART  was applEML4-ALK fusion gene [PML-RARα fusion gene [Actual cancer cell–derived RNA-Seq data for the identification of f-circRNA were obtained from the BioProject database (accession IDs PRJNA350335 and PRJNA315254). Whereas PRJNA350335 includes sequencing information on 9 lung cancer samples of H3122 cell line harbouring the ion gene .The “Fcirc” analysis pipeline includes 5 major steps Fig. , and theReads were aligned to a reference transcriptome with HISAT2  using deEML4-ALK and NPM1-ALK, EML4 and NPM1 were included in the same set (U) while ALK was included in the independent set (V).Gene pairs of known fusion genes were manually curated from multiple databases, including COSMIC , ChimerDEML4 with the same FLAG and CIGAR 40S60M values and to ALK with FLAG 4 and CIGAR 40M60S, this suggested that 1 segment of EML4 and the rest from ALK were on the same strand.In the next step, the unaligned reads were independently re-aligned to 2 sets of fusion gene sequences with low penalty. We decreased the maximum and minimum penalty for soft-clipping  and minimum alignment score . Other scoring parameters were set as default. After this re-alignment, reads with partial sequence alignment to fusion genes were selected. For single-end RNA-Seq data, reads without a FLAG value of 4 (-F 4) and for paired-end RNA-Seq data, reads with a FLAG value of 4, not 8, or 8, not 4 or 12 in the SAM file were selected (-f 4 -F 8 or -F 4 -f 8 or -f 12), respectively, ensuring that ≥1 read segment was aligned. Reads with paired chiastic clipping (PCC) signal were defined as fusion-related reads. For instance, if a segment of a read was simultaneously aligned to In the next step, the fusion breakpoint was determined. Therefore, we assumed that fusion-related reads were more likely to cover the respective fusion breakpoint and inferred the exact location from the majority of junction-supported reads. Subsequently, the fusion gene sequence was reconstructed around this predicted fusion breakpoint and the alignment of reads was recalibrated by re-aligning reads to the reconstructed fusion gene with low penalty. To evaluate our assumption, that fusion-related reads uniformly covered the fusion breakpoint, they were split into 2 groups (left and right fragments) in relation to the respective breakpoint. Then, the Wilcoxon signed-rank test was used to evaluate the read distribution by comparing the length of the left and right fragments.Circular RNAs transcribed from fusion genes were detected by searching for back-spliced reads. To improve the alignment of back-spliced reads with the reconstructed fusion gene, we changed the order of aligned and unaligned segments of some back-spliced reads to transform back-spliced reads to forward-spliced reads. The transformed reads were then re-aligned to the reconstructed fusion gene to evaluate whether they were truly back-spliced. Those reads covering a back-spliced junction indicated that they were attributable to f-circRNA.The resulting output format of Fcirc consists of tables of fusion linear transcripts and of f-circRNAs as well as SAM files for easier visualization of read distribution on the respective fusions.F-measure, which were defined as follows: precision=TP/(TP+FP); recall=TP/(TP+FN); F=2*/. TP, FP, and FN represent the true-positive, false-positive, and false-negative results, respectively. The F-measure simultaneously considers the effect of precision and recall. We also evaluated the number of supporting reads that were identified by the different methods and which reflect the ability to robustly detect the gene fusion. The final benchmark was the required computing time assuming a computational environment based on Ubuntu Linux with Intel Xeon E5-2620 v4 CPU at 2.10 GHz. Four CPU cores were used for each tool, and the running parameters for each tool are shown in Suppl. Fcirc and 6 previously published fusion detection methods Suppl. , includiF-measure) of Fcirc with those of other methods, we took advantage of RNA-Seq data from spike-in experiments, which included 9 synthetic cancer-associated fusion genes  [F-measures (0.86), indicating a better performance for balancing precision and recall  . Fcirc aall Fig. . Finallyall Fig. .In the next step, we calculated the number of fusion-supporting junctional reads for the different methods with respect to 10 different spike-in concentrations of the pooled synthetic gene fusion RNAs . Fcirc (red squares) not only identified the highest number of supporting reads but also had a very high accuracy for different spike-in concentrations, indicated by the increasing number of identified supporting reads of a given gene fusion Fig. . The NTRF-measures (0.91) in all of the simulated data, followed by Arriba (0.86)  than the other methods, with a high recall (85.64%) that was only second to Arriba (86.36%) Fig. . Fcirc a86) Fig. , and bot86) Fig. . For the86) Fig.  and pair86) Fig. , Fcirc rEWSR1-FLI1, (B) EML4-ALK, (C) PML-RARα, (D) KMT2A-MLLT3.To evaluate the ability of Fcirc to identify f-circRNA, we designed reads of 8 fusion circRNAs according to previous reports and plugged them into RNA-Seq data from normal pulmonary microvascular endothelial cells. We designed 2 types of RNA-Seq data: (i) a control dataset that contained only linear fusion transcripts and (ii) a dataset that contained pooled linear/circular fusion transcripts. Furthermore, single-end and paired-end RNA-Seq data, as well as different sequencing coverages  and read lengths (50 and 100 bp), were simulated. In paired-end samples, Fcirc successfully detected all 8 types of f-circRNAs from RNA plugged with pooled linear/circular fusion transcripts, whereas—as expected—no f-circRNAs were detected when only the linear fusion transcripts were present Fig. . The FciEML4-ALK fusion (BioProject ID PRJNA350335) and of various acute leukemia samples with PML-RARα fusion gene (BioProject ID PRJNA315254). We applied the Fcirc algorithm to the PRJNA350335 dataset in order to detect linear and circular fusion transcripts. Fcirc successfully identified EML4-ALK fusions in all 9 H3122 samples at the specific previously reported fusion breakpoint  than most other tools, being second only to the STAR-SEQR algorithm (0.889) (Suppl. KMT2A-MLLT3 (MLL-AF9) and PML-RARα fusions in the BioProject dataset PRJNA315254 [PML-RARα fusion  achieved the highest precision compared to other methods Suppl. , whereas) Suppl. . We deten Suppl. , with thEML4-ALK–rearranged NSCLC, shows that proteins encoded by fusion transcripts can also be harnessed as innovative drug targets [Fusion linear and circular transcripts (f-circRNA) are RNAs that are derived from rearranged genome translocations . Even th targets . This emTherefore, we developed the Python-based pipeline “Fcirc” to overcome these drawbacks and to enable researchers to accurately identify and quantify linear and circular (f-circRNAs) fusion transcripts from RNA-Seq data. Fcirc differs from other published fusion detection tools such as Arriba , ChimeraF-measures  , which warrants further investigation and biological characterization  Suppl. . These rn Suppl. .In conclusion, our study provides an insightful comparison of different fusion detection tools and suggests Fcirc as a powerful tool to detect linear and circular RNA transcripts of known fusion genes with high specificity in RNA-Seq datasets. Fcirc's reduced computing time will expedite the analysis of very large data sets and therefore improve our future understanding of the impact of gene fusion–related transcripts on cancer biology.Project name: Fcirc: A Comprehensive Pipeline for the Exploration of Fusion, Linear and Circular RNAshttps://github.com/WangHYLab/fcircProject home page: Operating system(s): Ubuntu 16.04/18.04, MacOSProgramming language: PythonOther program requirements: hisat2, samtools, numpy, scipy, pysamLicense: Massachusetts Institute of Technology https://bio.tools/Fcirc)Bio.tools id: biotools: Fcirc  designed for simulated RNA-Seq datasets.Supplemental Figure S2. Computing times of Fcirc in simulated single-end RNA-Seq data.Supplemental Figure S3. Performance comparison of different gene fusion detection tools in actual RNA-Seq data (BioProject ID PRJNA350335).Supplemental Table S1. Artificially designed fusion transcripts in simulated data (genome version: hg38).Supplemental Table S2. Artificially designed f-circRNAs in simulated data (genome version: hg38).Supplemental Table S3. Characteristics of fusion transcript detection tools.Supplemental Table S4. Summary of running parameters of 7 fusion detection tools.Supplemental Table S5. Precision and recall for synthetic RNA-Seq data.Supplemental Table S6. F-measure for synthetic RNA-Seq data.Supplemental Table S7. Computing time for synthetic data.Supplemental Table S8. Precision and recall for simulated RNA-Seq data.Supplemental Table S9. F-measure for simulated RNA-Seq data.Supplemental Table S10. Computing time for paired-end simulated RNA-Seq data.Supplemental Table S11. EML4-ALK fusions identified in actual cancer cell–derived RNA-Seq data (BioProject ID PRJNA350335).Supplemental Table S12. F-circRNAs identified by Fcirc in actual cancer cell–derived RNA-Seq data.Supplemental Table S13. Precision, recall, and F-measure for actual cancer cell–derived RNAseq data (BioProject ID PRJNA350335).Supplemental Table S14. Fusion genes identified in actual cancer cell–derived RNA-Seq data (BioProject IDs PRJNA350335 and PRJNA315254).giaa054_GIGA-D-19-00383_Original_SubmissionClick here for additional data file.giaa054_GIGA-D-19-00383_Revision_1Click here for additional data file.giaa054_GIGA-D-19-00383_Revision_2Click here for additional data file.giaa054_Response_to_Reviewer_Comments_Revision_1Click here for additional data file.giaa054_Response_to_Reviewer_Comments_Original_SubmissionClick here for additional data file.giaa054_Reviewer_1_Report_Original_SubmissionWilfried Haerty -- 12/1/2019 ReviewedClick here for additional data file.giaa054_Reviewer_1_Report_Revision_1Wilfried Haerty -- 3/9/2020 ReviewedClick here for additional data file.giaa054_Reviewer_2_Report_Original_SubmissionPieter-Jan Volders -- 12/3/2019 ReviewedClick here for additional data file.giaa054_Supplemental_FilesClick here for additional data file.bp: base pairs; circRNA: circular RNA; COSMIC: Catalogue of Somatic Mutations in Cancer; f-circRNA: fusion-circular ribonucleic acid; FDA: Food and Drug Administration; FN: false negative; FP: false positive; NCBI: National Center for Biotechnology Information; NSCLC: non–small cell lung cancer; PCC: paired chiastic clipping; RNA-Seq: RNA sequencing; SAM: Sequence Alignment/Map format; SRA: Sequence Read Archive; TP: true positive.The authors declare that they have no competing interests.H.W. conceived the hypothesis. Z.C., H.X., Y.X., X.C., Y.D., and J.Z. designed and performed the pipeline workflow and the analyses. H.W., J.K., H.X., and Z.C. interpreted the results and wrote the manuscript.This work was supported by grants from the National Natural Science Foundation of China (31771469 and 31571363 to H.W.), a grant from the National Key Research and Development Program (2017YFC0908500 to H.W.), and a Mildred-Scheel postdoctoral fellowship from the German Cancer Aid Foundation(70111755 to J.K.).


The RJunBase aims to integrate and characterize all RNA splice junctions of both healthy or pathological human cells and tissues. This new database facilitates the visualization of the gene-level splicing pattern and the junction-level expression profile, as well as the demonstration of unannotated and tumor-specific junctions. The first release of RJunBase contains 682 017 linear junctions, 225 949 back-splice junctions and 34 733 fusion junctions across 18 084 non-cancerous and 11 540 cancerous samples. RJunBase can aid researchers in discovering new splicing-associated targets and provide insights into the identification and assessment of potential neoepitopes for cancer treatment.Splicing is an essential step of RNA processing for multi-exon genes, in which introns are removed from a precursor RNA, thereby producing mature RNAs containing splice junctions. Here, we develope the RJunBase ( In the ll lines . Additio cancers . Ryan et) tumors . AlthougIn this study, we developed a web-accessible database, named RJunBase, to provide information on three types of RNA splice junctions that are derived from RNA-seq data of various non-cancerous and cancerous samples. This tool aims to help users search for their desired targets and obtain information about their splice patterns and expression profile. The whole dataset was designed to be easily browsed, queried, and downloaded through its webpage. In addition, RJunBase is expected to allow researchers submit new splice junctions and their expression profiles of human samples.https://www.gencodegenes.org). Linear junctions were matched onto gene coordinates using the genomic intersection functionality of BEDTools (https://bedtools.readthedocs.io/) and were further classified as intragenic or intergenic junctions. All junctions were annotated with gene information, and junctions in the intergenic region were annotated with information of the downstream gene. In addition, oncogene and tumor suppressor gene annotations were derived from ONGene  portal . Metastam ONGene  and TSGem ONGene , and wer  contained the canonical splice site (GT-AG).The junctions were identified as tumor-specific junctions when the following criteria were met: NT_max = 0 & nTE_max = 0 & Freq_tumor > 5%; or Tumor_median > 10 * NT_max & Tumor_median > 10 * nTE_max & Freq_tumor > 5%. Tumor_median is the median expression value of tumor samples, NT_max is the maximal expression value of TCGA normal samples, nTE_max is the maximal expression value of GTEx normal tissues, and Freq_tumor is the frequency of junctions expressed in the cancer.https://mioncocirc.github.io/) (The location and expression information of back-splice junctions from over 2000 samples across 27 cancer types were obtained from MiOncoCirc database (hub.io/) . The totet al.  to detect gene fusions emerging at annotated exon boundaries. This algorithm used RNA-seq fastq files from 9946 TCGA samples as input data and identified over 30 000 fusions. Gene information for the two types of junctions in the RJunBase was then annotated based on the GENCODE database (release 29). The expression of fusion junctions were standardized as linear junctions according to: raw_counts/sum_cov × 10 000 000 (sum_cov: sum coverage of samples). The expression frequency (and corresponding sample number) and median value in cancer cohorts were calculated for each back-splice and fusion junction.Fusion junctions were downloaded from a previous report published by Dehghannasiri et al. . They deWe used a custom python-based script to plot all relevant forward- and back-splice junctions for each gene. Hosting genes of fusion junctions were efficiently and flexibly visualized in a circular layout using the R software package ‘circlize’ , and theet al.  and Akka 2.6.3 (web server). All data were handled and organized into a MySQL Database Management System (version 5.7). The interface of the website was designed and achieved using the Twirl template engine. The query system for the database was based on Scala script. The website was tested in several distinct popular web browsers such as Google Chrome, Firefox, and Internet Explorer.RJunBase collects and integrates three types of splice junctions from different non-cancerous and cancerous human tissues Figure . The curUsers can browse the information on junctions by clicking the ‘Browse’ tab on the left-side navigation menu, and the browse page will present the entry point to tables containing all the information on the three types of junctions Figure . Users cTo obtain the analysis results only a click on the ‘Run’ button is necessary and a dynamic tabular with optional boxes will be displayed, in which each row and column show a junction and region-specific information respectively, including Junction ID, Junction location, Junction type, Gene symbol, Gene type, Tumor median, Tumor frequency (sample number), NT frequency (sample number), Tumor specific types, Annotation. Users can also obtain Transcript ID, NT median, Normal median, Normal frequency (sample number), and Alternative splicing types by clicking the corresponding menu options Figure . By defaERBB2) or more gene symbols that are separated by commas  in the ‘Keyword Search’ field. Advanced filters, including Junction type, Annotation, Tumor-specific types, and Genomic location, allow users to further filter the results . RJunBase uses the Kaplan–Meier method for hypothesis evaluation, and the thresholds for high/low expression cohorts are adjustable.ETV4) holds a potential role in endometrial tumor-specific estrogen receptor binding  and mutations in cis-regulatory sequences or trans-acting splicing regulatory factors ,28. For We also discovered plenty of unannotated novel junctions in pan-cancer tissues and non-cancerous tissues, which could be divided into intergenic and intragenic junctions. The intergenic junctions may be generated by epigenetic alterations such as transcriptional activation, whereas the intragenic junctions may be produced by aberrant splicing in a single gene. It should be noted that some unannotated junctions may be by-products of the bioinformatics RNA splicing analysis. Recent advances in long-read sequencing identified full-length transcripts in different species . By usinThe RJunBase will be regularly updated to continuously include more splice junctions of healthy and pathological human tissues, especially cancer tissues, including rare cancers. We believe that RJunBase will serve as an important resource and a powerful tool for studying splice junctions in human cancer transcriptomes and further understanding specific transcript production. Furthermore, the use of this database could potentially lead to the discovery of diagnostic and therapeutic targets for cancer.gkaa1056_Supplemental_FileClick here for additional data file.


Drug-induced liver injury (DILI) is an adverse reaction caused by the intake of drugs of common use that produces liver damage. The impact of DILI is estimated to affect around 20 in 100,000 inhabitants worldwide each year. Despite being one of the main causes of liver failure, the pathophysiology and mechanisms of DILI are poorly understood. In the present study, we developed an ensemble learning approach based on different features  to predict drugs that might cause DILI and gain a better understanding of the mechanisms linked to the adverse reaction.We searched for gene signatures in CMap gene expression data by using two approaches: phenotype-gene associations data from DisGeNET, and a non-parametric test comparing gene expression of DILI-Concern and No-DILI-Concern drugs (as per DILIrank definitions). The average accuracy of the classifiers in both approaches was 69%. We used chemical structures as features, obtaining an accuracy of 65%. The combination of both types of features produced an accuracy around 63%, but improved the independent hold-out test up to 67%. The use of drug-target associations as feature obtained the best accuracy (70%) in the independent hold-out test.When using CMap gene expression data, searching for a specific gene signature among the landmark genes improves the quality of the classifiers, but it is still limited by the intrinsic noise of the dataset. When using chemical structures as a feature, the structural diversity of the known DILI-causing drugs hampers the prediction, which is a similar problem as for the use of gene expression information. The combination of both features did not improve the quality of the classifiers but increased the robustness as shown on independent hold-out tests. The use of drug-target associations as feature improved the prediction, specially the specificity, and the results were comparable to previous research studies.The online version contains supplementary material available at 10.1186/s13062-020-00288-x. Drug safety is one of the main reasons of drug attrition during development , 2. Althin-silico methods have been proposed to predict hepatotoxicity of drugs. Among these, machine learning models trained using drug structural features have shown a good accuracy [Sumsion et al. [Chierici et al. [Sumsion et al. [Chierici et al. [Several accuracy –10. Furtaccuracy . With thaccuracy , 13. Sumn et al.  evaluaten et al.  and Chiei et al. ) composei et al.  , which identifies the fingerprints of the SMILES and computes the fingerprint similarity between pairs of SMILES. The full list of SMILES is provided in Supplementary Table The chemical structures of the drugs considered in the study were provided by the CAMDA challenge in the form of Simplified molecular-input line-entry system (SMILES) string. In order to use this type of data, we calculated the similarity between all compounds, creating a matrix of chemical similarity. Specifically, we used the R package RxnSim  to calcuRxnSim . We usedThe targets of the compounds considered in the study were retrieved from three different databases: DGIdb , HitPickcaret [We created a supervised machine learning pipeline  and 51 labelled as no DILI. To create a balanced testing dataset, as there were less drugs labelled as no DILI, we randomly picked the 30% of the 51 no DILI drugs (15 drugs), and the same number of DILI drugs, maintaining the ratio of Most-DILI-Concern (29.8%) and Less-DILI-Concern (70.2%): 4 Most-DILI-Concern drugs (the 29.8% of 15) and 11 Less-DILI-Concern drugs (the 70.2% of 15). The rest of the drugs (109 DILI drugs and 36 no DILI drugs) were used for creating multiple training datasets. In order to have balanced training datasets, while at the same time, to cover as many DILI drugs as possible, we created 10 different training datasets. All of them have the same 36 no DILI drugs , but each of the training dataset has a different subset of DILI drugs. Accordingly, among the 109 DILI drugs, we picked randomly 11 Most-DILI-Concern drugs (29.8% of 36) and 25 Less-DILI-Concern (70.2% of 36)  that indicate the quality of the model. Lastly, the testing set predictions of the 10 models were used as features to train a random forest classifier that combined them into a final model. The final model was used to classify the drugs of the independent hold-out test dataset into DILI drugs and non-DILI drugs.The 10 training datasets were used to train 10 different models. For each model, the hyperparameters of the machine learning classifier were tuned using the functions ge caret . SpecifiCMap collects gene expression signatures obtained from cell lines upon treatments with different drug concentrations and durations. The treatment dose ranges from the drug’s reported effective concentration, if known, to a relatively high concentration of 10 μM or more, often adopted in high-throughput cell based screens . In ordeAs an initial exploratory analysis of the training data set, we analyzed the transcriptional response of the different drugs using k-nearest neighbor clustering algorithm   [Once defined the set of genes for the different DILI-related phenotypes as annotated in DisGeNET, we retrieved their gene expression data from the CMap L1000 Assay Platform. For each DILI-related phenotype, we trained an independent machine learning model using the expression levels of their genes as features. The average accuracy obtained for the models of all DILI-related phenotypes is 57% in the testing set. This means that for some specific phenotypes the accuracy was higher than 57%. Therefore, we inspected the results for all phenotypes separately, observing those with higher accuracy than others Fig. . The phe. (2019) , where 1Our current knowledge of genotype-phenotype associations is still incomplete and therefore we might miss relevant genes associated to DILI. It has been demonstrated that the products of disease-associated genes tend to be highly connected in the protein-protein interaction network, forming the so-called disease modules , 36. BasUsing the network-based prioritization web server GUILDify , we exteSection 2, obtaining similar values to that of when using the 978 landmark genes albeit with a slightly higher specificity  and used the expression level of these genes as input feature to the machine learning classifier. As shown in Fig. P-value lower than 0.05, obtaining a gene signature composed of 66 genes (referred from now on as DILI landmark gene signature) . For each gene, we checked whether the expression values were significantly different between DILI and No-DILI-Concern drugs. Therefore, for each landmark gene, we applied a two-sided Wilcoxon test, a non-parametric test comparing the expression of the gene in the samples of DILI-Concern drugs and No-DILI-Concern drugs. We selected the genes with a Consistent with the known heterogeneity of transcriptomics response in hepatotoxicity, the genes in the identified signature were typically perturbed only in a small subset of the samples, failing to represent a common response that could be explained by gene expression changes Fig. . HoweverPeng et al. (2019) [Peng et al’s study. Therefore, although the approach succeeded at improving the predictive capacity of the classifiers, the results could lead to overfitting by the available data, as the gene signature is not related to any specific biological function.The functional enrichment analysis of the 66 genes did not yield specific functions significantly associated with the genes. This indicates that, even though the selected genes improve the capacity of the classifiers to predict DILI-causing drugs in comparison with using all the landmark genes, they are not related with specific biological processes. Additionally, we compared these 66 genes to the 145 identified by . (2019)  as assocBesides using gene expression data, we investigated the incorporation of orthogonal information derived from chemical structure of the drug and its targets. The chemical structure and the molecular descriptors of the drugs have already been used in machine learning models, showing a fair predictive capacity . Here, wEventually, we obtained a higher prediction accuracy when combining chemical structure and CMap than when applying them separately Fig. . When usNext, we explored the use of drug-target associations as a feature to predict DILI. We considered any drug-protein pair that had been reported in DGIdb  or prediIn the previous sections, when using CMap gene expression data, we selected only the samples from the PHH cell line with 10 μM dose and treatment duration of 24 h. We focused on the drug response in liver cells. However, the data of CMap tends to have a high variation of expression between samples even for the same gene. Therefore, to avoid biases caused by the use of unrelated samples, we experimented using only the top correlated samples for each drug. This consists in computing the correlation between all the samples exposed to a drug  and selecting the ones that are more correlated between themselves. We selected the pairs of samples from different cell lines that have a Pearson correlation above 0.5, or otherwise we kept the pair that was more correlated. To use a correlation threshold of 0.5 guarantees that the expression of the samples selected is consistent enough across several cell lines. Once the correlated samples are selected, we use the median gene expression as feature. Although the approach was theoretically promising, the prediction accuracies with the use of correlated samples are generally worse than using specific conditions, obtaining MCC values ranging from − 0.12 to 0.21  alone and in combination with drug-centric information in the form of structural similarity (Tanimoto scores) and protein targets , the intrinsic data noise from the CMap dataset is increased as well, hence hampering the prediction. Still, we think that using protein-protein interactions to extend our information on DILI targets and hepatotoxicity-associated genes without using gene expression data could be an interesting feature to explore in the future.After working with transcriptomics data from CMap, we observed variability of the results depending on the pre-processing of the samples. We tried two different strategies that led to different results: (i) focusing on samples from a unique cell line and dose-time point for each drug, and (ii) selecting the most correlated samples for each drug. This is by no means comprehensive and various possible strategies such as using other cell lines and dose-time points, or discarding the samples with low correlation between replicas (‘distil_cc_q75’ < 0.2) and selecting the sample with highest transcriptional activity score  could beWhen focusing on samples from a unique cell line and dose-time point, we decided to use the highest concentration and the longest treatment duration. In this way, we were including perturbations possibly leading to adversities for the adaptive mechanisms of the cells. We acknowledge that focusing on increased exposure of the drug to characterize DILI is a relatively strong assumption as there could be certain compensatory mechanisms kicking in after a while depending on the specific compound and cell line. Nevertheless, we think that employing the highest dose at the longest time of exposure is likely to be a fair representation of the effect of DILI in the cells after the administration of the drug.Apart from the limitations inherent in the CMap dataset, we detected: (i) an important genetic diversity between the diverse DILI-related phenotypes from DisGeNET , thus a global predictor categorizing drugs as simply DILI vs no-DILI might not be realistic. And, third, there is substantial heterogeneity in the transcriptomics data from CMap. There is also variation between the results of the testing set and the independent hold-out test set, that could be caused by the latter being unbalanced (as the labels remain hidden). Still, the variation between the machine learning algorithms (random forest vs gradient boosting machine) is not appreciable in most cases . We observed that selecting a specific gene signature either using phenotype-gene associations data (DisGeNET) or a non-parametric test (Wilcoxon test) of differential expression between DILI and no-DILI samples constituted a better feature than the whole landmark genes in CMap. However, the accuracy of the best performing classifier is around the 70% mark , stating the limitations of predicting DILI. The results are very similar to previous publications –10, 12. Additional file 1.Additional file 2.Additional file 3.Additional file 4.Additional file 5.Additional file 6.Additional file 7.Additional file 8.Additional file 9.


Breast cancer is a collection of multiple tissue pathologies, each with a distinct molecular signature that correlates with patient prognosis and response to therapy. Accurately differentiating between breast cancer sub-types is an important part of clinical decision-making. Although this problem has been addressed using machine learning methods in the past, there remains unexplained heterogeneity within the established sub-types that cannot be resolved by the commonly used classification algorithms.In this paper, we propose a novel deep learning architecture, called DeepTRIAGE , which uses an attention mechanism to obtain personalised biomarker scores that describe how important each gene is in predicting the cancer sub-type for each sample. We then perform a principal component analysis of these biomarker scores to visualise the sample heterogeneity, and use a linear model to test whether the major principal axes associate with known clinical phenotypes.Our model not only classifies cancer sub-types with good accuracy, but simultaneously assigns each patient their own set of interpretable and individualised biomarker scores. These personalised scores describe how important each feature is in the classification of any patient, and can be analysed post-hoc to generate new hypotheses about latent heterogeneity.We apply the DeepTRIAGE framework to classify the gene expression signatures of luminal A and luminal B breast cancer sub-types, and illustrate its use for genes as well as the GO and KEGG gene sets. Using DeepTRIAGE, we calculate personalised biomarker scores that describe the most important features for classifying an individual patient as luminal A or luminal B. In doing so, DeepTRIAGE simultaneously reveals heterogeneity within the luminal A biomarker scores that significantly associate with tumour stage, placing all luminal samples along a continuum of severity. Breast cancer is a collection of multiple tissue pathologies with a joint genetic and environmental aetiology, and is a leading cause of death among women worldwide. During the progression of cancer, inherited or acquired mutations in the DNA change the sequence (or amount) of the messenger RNA (mRNA) produced by the cell, thereby changing the structure (or amount) of functional protein. As such, mRNA can serve as a useful proxy for evaluating the functional state of a cell, with its abundance being easily measured by microarray or high-throughput RNA sequencing (RNA-Seq). Indeed, mRNA abundance has already been used as a biomarker for cancer diagnosis and classification , 2, cancp≫n problem that is usually addressed by feature selection or feature engineering  be the representation of sample j in this space. Our goal is to train a binary classifier that learns whether sample j belongs to the LumA class (yj=1) or the LumB class (yj=0).Let Ed×m be an embedding matrix and let ei= be the embedding vector for feature i∈{1:d}. Using Eq. m-dimensional embedded vector of feature i for sample j:Letlet ei=ei,…,eim befe, parametrised by Θe, is the function that captures the relationship between the scalar value xji and the embedding vector ei. Note that the same embedding matrix is used for all samples.where j using its embedded vectors, as shown in Eq. Now, we can define a new representation for sample p-dimensional representation of the raw feature data, Θx, and βji is a scalar value denoting the individualised importance that feature i has in the classification of sample j. Note that m≪d and usually p≪m. In other words, the dimensionality of the embedding space is much smaller than the dimensionality of the raw feature space, and the dimensionality of the final representation βji as the attention of sample j to feature i and compute it using Equations where entclass1pt{minimaΘα. Equation where j as fy for LumA vs. LumB classification:Now, given the fixed size representation of sample Θyis the set of parameters for fy.where E, we have defined our model through Equations Given the embedding matrix E. For instance: end-to-end learning with an unsupervised component added to the model, estimation using auto-encoders, or dimensionality reduction using PCA. We chose to use random vectors because it has been shown that their performance is comparable with the aforementioned techniques , where βji is the individualised importance score for sample j and feature i, we can construct an importance score matrixB by stacking βj for all samples.Given NMF) and principal component analysis (PCA) of the importance score matrix B. As a point of reference, we also perform an ordination of the raw feature space from “To detect emerging patterns within the individualised importance scores, we perform non-negative matrix factorisation  annotation set  gene set. Figure GO terms, offering another perspective into the structure of the importance score matrix.Figure Both visualisations show that the kinetochore organisation gene set can meaningfully discriminate between the luminal A and luminal B cancer sub-types. This gene set contains 5 members: SMC4, NDC80, SMC2, CENPH, and CDT1. Figure and describing heterogeneity within the cancer sub-types. While Fig. among the luminal A samples only, reveals that small values in the first axis (PC1) significantly associate with the lower T stages, while large values significantly associate with the N2 stage (p<0.05). Meanwhile, large values in the second axis (PC2) significantly associate with the T4 stage (p<0.05). This suggests that the luminal A samples which are closest to luminal B samples in the PCA tend to be worse tumours. This is consistent with the literature which describes luminal B cancer as a more severe disease [GO annotation features  annotation set which organises genes according to canonical functional pathways  explains much of the inter-group variability  significantly associate with the T1 stage (p<0.05). This suggests that the heterogeneity uncovered by the DeepTRIAGE architecture places patients along a diagnostically-relevant continuum of tumour severity. Again, this level of resolution is not provided by other machine learning algorithms and is not evident in the ordination of the unattended annotation-level data (see Figure The NMF and PCA ordination of the see Fig. . This isBreast cancer is a complex heterogeneous disorder with many distinct molecular sub-types. The luminal breast cancer class, comprised of the luminal A and luminal B intrinsic sub-types, varies in disease severity, prognosis, and treatment response , and hasand describe the heterogeneity within and between cancer sub-types. While commonly used feature selection methods prioritise features at the population-level during training, our attention mechanism prioritises features at the sample-level during testing. By applying DeepTRIAGE to the gene expression signatures of luminal breast cancer samples, we identify canonical cancer pathways that differentiate between the cancer sub-types and explain the variation within them, and find that some of this intra-class variation associates with tumour severity.In developing DeepTRIAGE, we also innovate the attention mechanism so that it extends to high-dimensional data where there are many more features than samples. Using DeepTRIAGE, we show that the attention mechanism can not only classify cancer sub-types with good accuracy, but can also provide individualised biomarker scores that reveal


Predictive patient stratification is greatly emerging, because it allows us to prospectively identify which patients will benefit from what interventions before their condition worsens. In the biomedical research, a number of stratification methods have been successfully applied and have assisted treatment process. Because of heterogeneity and complexity of medical data, it is very challenging to integrate them and make use of them in practical clinic. There are two major challenges of data integration. Firstly, since the biomedical data has a high number of dimensions, combining multiple data leads to the hard problem of vast dimensional space handling. The computation is enormously complex and time-consuming. Secondly, the disparity of different data types causes another critical problem in machine learning for biomedical data. It has a great need to develop an efficient machine learning framework to handle the challenges.In this paper, we propose a fast-multiple kernel learning framework, referred to as fMKL-DR, that optimise equations to calculate matrix chain multiplication and reduce dimensions in data space. We applied our framework to two case studies, Alzheimer’s disease (AD) patient stratification and cancer patient stratification. We performed several comparative evaluations on various biomedical datasets.In the case study of AD patients, we enhanced significantly the multiple-ROIs approach based on MRI image data. The method could successfully classify not only AD patients and non-AD patients but also different phases of AD patients with AUC close to 1. In the case study of cancer patients, the framework was applied to six types of cancers, i.e., glioblastoma multiforme cancer, ovarian cancer, lung cancer, breast cancer, kidney cancer, and liver cancer. We efficiently integrated gene expression, miRNA expression, and DNA methylation. The results showed that the classification model basing on integrated datasets was much more accurate than classification model basing on the single data type.The results demonstrated that the fMKL-DR remarkably improves computational cost and accuracy for both AD patient and cancer patient stratification. We optimised the data integration, dimension reduction, and kernel fusion. Our framework has great potential for mining large-scale cohort data and aiding personalised prevention. Patient stratification has widespread biomedical and clinical applications, including diagnosis, prognosis, and treatment response prediction. A clinically useful prediction algorithm should be accurate, generalizable, be able to integrate diverse data types, and handle sparse data –5. To acAlzheimer’s disease is a neurological disorder in which the death of brain cells causes memory loss and cognitive decline. Aging is the primary cause of AD, however, there are several other reasons related to lifestyle, such as physical inactivity, obesity, unhealthy diets, alcohol abuse, etc. . Among AMagnetic Resonance Imaging (MRI) data has been popularly used in AD patient stratification due to MRI’s high-quality three-dimensional images of brain. Based on MRI data, regions of interest (ROI) which affect disease development could be revealed, contributing significantly to AD diagnosis and treatment. There are two main approaches based on ROIs, the single-ROI based approach  and the Cancer is not only threatening but also very diverse. Cancer patient stratification has been one of most challenging topics in biomedical informatics. Previous work have either focused on a specific data type of interest, such as gene expression, DNA methylation –20 or coThe computation for learning high dimensional space is extremely complex and time-consuming. Since data types are of great difference , it is therefore essential to unify data measurement before integrating them. Lin et al. in  have proIn this paper, we have proposed a novel computational framework based on fast multiple kernel learning and dimension reduction (fMKL-DR in short), addressing challenges in AD and cancer patient stratification. In the case study of AD patients, we enhanced significantly the performance of the multiple-ROIs approach when comparing with previous methods. Accuracy and Area Under a Curve (AUC) demonstrated that our proposed method was more accurate and robust than previous work. In the case study of cancer patients, the framework was applied to six types of cancers, i.e., glioblastoma multiforme cancer, ovarian cancer, lung cancer, breast cancer, kidney cancer, and liver cancer. We efficiently integrated gene expression, miRNA expression, and DNA methylation. The results showed that the classification model basing on integrated dataset was much more accurate than classification model basing on the single data type. The results obtained by both AD and cancer applications manifest that our developed model potentially stratifies patients and later aid disease prevention and prognosis.The proposed framework is demonstrated in Fig. In this section, we present the methods for collecting and pre-processing MRI images and genomics data for Alzheimer’s disease; proteomics and genomics data for cancer diseases.MRI images of Alzheimer’s disease patients were extracted from Alzheimer’s Disease Neuroimaging Initiative (ADNI).2Gk corresponding to six measures, namely as GCGMV, GCT, GCSA, GCC, GCFI, GSV. In a graph Gk, the set Vk denotes set of vertices (vi), representing the regions. We denote Ek as the set of edges (eii), consisting the weighted connections between two regions. The weight wij of the edge eij between two regions i and j is calculated as following:dij is the distance between two regions and dij =  ∣ mi − mj∣ . For example, GCGMV consists of 78 vertices and 3003 edges. In case of the sub-cortical region, GSV was constructed by the set Vi of 12 vertices and the set Ei of 66 edges. Both Vi and set Ei are matrices R, where m is the number of samples/subjects in our model, and n is the number of vertices or the number of edges, respectively. The illustration of the AD’s MRI images pre-processing is shown in Fig. After calculating the six measures for cortical and sub-cortical regions, we represented the obtained data in terms of graphs. We generated six graphs entclass1pt{minimaIn addition to MRI images, gene expression data of AD patients is very interesting for stratifying AD patients. From the ADNI database,R, in which columns are subjects, and rows are genes.We extracted six cancer patient datasets from the TCGA database The values in each dataset have different magnitudes, units, and ranges. This variety causes an issue that the higher magnitude dataset will have greater weight than the lower ones. Therefore, we rescaled all of data values into the same range, enabling the equity between datasets. We used min-max scaling to scale the data into the range .Multiple Kernel Learning (MKL) is a machine learning method, modelling a kernel ensembled from many kernel functions or kernel matrices. Recent research on MKL have shown that learning SVMs with multiple kernels not only increases the accuracy but also enhances the expandability of the classification . The MKLIn Lin et al. , multiplN), the number of data types (M), and the dimensions after being reduced (P). In case that the value of M is small, often between 3~10, the number of dimensions after being reduced is small . Therefore, the computation complexity is O(N3), which is polynomial time. The MKL-DR training algorithm calculates iterative equations:There are three parameters that affect the performance of the MKL-DR, i.e., the number of samples , ,  calculatMatrix chain multiplication has a combinatory property, meaning that changing calculation order between a pair of the matrices will affect the number of product operations without modifying the multiplication results. Therefore, we propose to use a dynamic-programming-based procedure to find the calculation sequence that gives minimum product operations. If the number of product operations is minimum, the computation time of the equations will be reduced.The minimum product operation order problem based on dynamic programming:A1, A2, …, AN with size of Ai is di − 1 × di.Given N matrices Problem: Find the order of product matrix chain A1 × A2 × … × AN to minimize the number of product operations.Solution: Construct matrix F is a matrix N × N, with element F is the total product operations to calculate matrix chain multiplication from Ai to Aj (Ai × Ai + 1 × … × Aj). The formula to calculate F defined by:t = i + 1, i + 2, …, j − 1. In other words, t is a midpoint to insert the parentheses to change the calculation order so that the number of product operations is minimum:Ai × Ai + 1 × … × At) is di–1 × dt, and the dimension of the matrix of (At + 1 × At + 2 × … × Aj) is dt × dj.The dimension of the matrix of . However, in the above equations, N (the number of matrices in chain) is small and equals to 4. Consequently, the time consumption of MCMO is trivial. Moreover, MCMO only calls 2 times in fMKL-DR, which is built as demonstrated in Algorithm 2.The MCMO complexity is The fast MKL-DR algorithm is described below.OA and Oβ. These matrices have the same values as MKL-DR ones.The matrices We carried out three main experiments to investigate the performance of our method. The first experiment was done on the AD patients’ the MRI image dataset, comparing our proposed method to previous work. In the second experiment, we tested our method on the AD patients’ gene expressions. The last experiment was designed based on the six cancer patient datasets. In all of the experiments, we evaluated three measurements including accuracy, AUC, and computational time.k = exp(−‖x − x'‖2/2σ2). We run the experiments with 5 different parameters σ∈ {10− 6, 10− 3, 1, 103, 106} to generate 5 kernel matrices. We constructed 60 kernel matrices from 12 original datasets, that were used as input data for the next step.Step 1. We built multiple kernel matrices by using different parameters from 12 original datasets. Specifically, from each dataset, we used the Gaussian kernel function AD/NC, CAD/MCI, CNC/MCI, and CMCIc/MCInc correspondingly. We employed libSVM libraryStep 2. Develop four models for the four binary classification problems, AD and NC, AD and MCI, NC and MCI, and MCIc and MCInc, denoted by CAD/NC, CAD/MCI, CNC/MCI, and CMCIc/MCInc), 20 experiments were performed. In each experiments, 2/3 dataset were randomly selected from the original dataset for training, and the rest was used for testing. The best result among the 20 experiments is reported and statistically tested.Step 3. For each classification problems , CPolinomial with Polinomial kernel matrix), CLinear with Linear kernel matrix), CSigmoid with Sigmoid kernel matrix), and CfMKL-DR with unified kernel matrix that is got in Step 2). The libSVM parameters were set as default values (showed in Step 1 of E1).Step 3. Develop five classification models based on the libSVM library, i.e., CStep 4. Carry out the same procedure as Step 3 in E1.Step 5. Compare the results between the classification model using single gene expression dataset with different kernel functions (Step 1) and the unified kernel (Step 2).We designed the experiment based on AD patient gene expression dataset as follows:β. Both of initializations produce similar results, even though the first initializing A obtains faster convergence. Integrate multiple data sources and reduce the number of data dimensions by fMKL-DR.Step 1. Run the fMKL-DR algorithms by initializing A or GE for the gene expression dataset, CDNA for the DNA methylation dataset, CmiRNA for the mirRNA expression dataset), and CfMKL-DR for the unified kernel matrix obtained in Step 1. For the three models , we used the default values in libSVM with Gaussian kernel function . In the case of the model CfMKL-DR, the kernel is the one obtained unified kernel matrix and the other parameters are set as the default values in libSVMs.Step 2. Develop four classification models based on three single datasets and one combined dataset and SVMs, i.e., CStep 3. Perform the same procedure as Step 3 in E1.Step 4. Compare the results between the classification model using a single data type and the one with data integration.We carried out the following four steps on each dataset to evaluate the proposed method.These abovementioned experimental steps were run on the six patient datasets of cancer patients to evaluate the efficiency of the methods.t-test with n = 20. At 95% confidence level, the hypothesis tests (on mean of accuracy and AUC) were done to evaluate whether they are statistically significant.We performed statistical tests to assess the robustness of the obtained results, specifically one sample To investigate performance of our classification model, we carried out four experiments for four classification subgroups including AD/NC, AD/MCI, NC/MCI, MCIc/MCInc (MCI and converted to AD/ MCI and not converted to AD). We evaluated our method by comparing accuracy and area under curve (AUC) of our classification model to previous works, which used the same dataset of the MRI images from ADNI.Tables Figure k1 = Gaussians, k2 = Polynomial, k3 = Linear, k4 = Sigmoid, and our integrated kernel function  were removed. The missing mechanism will be deeply analysed to define the type of missing data, missing completely at random or missing at random or missing not at random. Depending on the types of missing data, a appropriate technique, such as data imputation, will be applied. Secondly, parameter optimisation will be of interest. We set the default values in our experiments to ensure the comparative evaluation with the other related work, however the better performance can be obtained by testing several sets of parameters and parameter optimisation algorithms.The proposed framework is very useful when analysing high dimensional genomics data. Dimension reduction can be applied in the significance analysis on gene expression data. In multi-assay data exploration, dimension reduction facilitates downstream gene set, pathway and network analysis of variables. Integrative data analysis is of great interest due to the complexity of biology and medicine. Because multi-omics data, such as DNA sequence, epigenome, transcriptome, protein, metabolites, is more and more available, there is an increasing need for developing methods with high performance. Single kernel methods are not suitable to encode heterogenous or multi-modal datasets. The proposed framework can be easily adjusted to ingrate new data types, making it flexible in other applications, such as biomarker identification. The framework is adaptable for other disease studies, in addition to cancer and AD. Precision medicine is required big data integration, especially electronic health records and whole genome sequencing data. Therefore, our method will be very beneficial to precision medicine.In this paper, we have proposed the accurate and fast kernel learning framework. We employed the framework to stratify AD patients and cancer patients. We handled a wide range of data, including the MRI image data for AD, the gene expression data for both AD and cancer, miRNA expression, DNA methylation for cancer. By carrying out a number of testing strategies, the results showed that our model performed better than previous work, in term of accuracy, AUC, and computational time. In fact, the electronic health records will be more and more available, offering better insights of patient studies. As there are more and more data available from diverse sources, it is emerging to develop computational method to combine and mine those heterogenous data. Our proposed framework is very promising to handle high dimensional data and to aid precision medicine.