Gene expression plays a key intermediate role in linking molecular features at the DNA level and phenotype. However, owing to various limitations in experiments, the RNA-seq data are missing in many samples while there exist high-quality of DNA methylation data. Because DNA methylation is an important epigenetic modification to regulate gene expression, it can be used to predict RNA-seq data. For this purpose, many methods have been developed. A common limitation of these methods is that they mainly focus on a single cancer dataset and do not fully utilize information from large pan-cancer datasets.Here, we have developed a novel method to impute missing gene expression data from DNA methylation data through a transfer learning–based neural network, namely, TDimpute. In the method, the pan-cancer dataset from The Cancer Genome Atlas (TCGA) was utilized for training a general model, which was then fine-tuned on the specific cancer dataset. By testing on 16 cancer datasets, we found that our method significantly outperforms other state-of-the-art methods in imputation accuracy with a 7–11% improvement under different missing rates. The imputed gene expression was further proved to be useful for downstream analyses, including the identification of both methylation–driving and prognosis-related genes, clustering analysis, and survival analysis on the TCGA dataset. More importantly, our method was indicated to be useful for general purposes by an independent test on the Wilms tumor dataset from the Therapeutically Applicable Research to Generate Effective Treatments (TARGET) project.TDimpute is an effective method for RNA-seq imputation with limited training samples. The recent development of molecular biology and high-throughput technologies facilitates the simultaneous measurement of various biological omics data such as genomics, transcriptomics, epigenetics, proteomics, and metabolomics for a single patient. Compared with single-omics analysis, integrative analyses of multi-omics data provide comprehensive insights into cancer occurrence and progression and thus strengthen our ability to predict cancer prognosis and to discover various levels of biomarkers. However, owing to limitations of experimental techniques and relatively high costs for measuring the multi-omics data, most samples are not measured with all types of omics data and lack 1 omics data type . This problem is prevalent in publicly available multi-omics datasets including The Cancer Genome Atlas (TCGA). Because gene expression affects clinical outcomes or phenotypes more directly than molecular features at the DNA level such as DNA methylation and genetic variants [When the missing data happens at random positions in single-omics data, the missing data can be imputed by traditional methods, such as singular value decomposition imputation (SVD) and k-nearest neighbor (KNN) . HoweverIn recent years, the deep neural network technique has demonstrated its superiority for modeling complex nonlinear relationships and enjoys scalability and flexibility. For gene expression imputation or prediction, many deep learning models have also been proposed. Chen et al. built a multilayer feedforward neural network to predict the expression of target genes from the expression of ∼1,000 landmark genes . With thIn such scenarios, transfer learning is usually considered as a promising method, in which parameters trained for a task with a large amount of data are reused as the initialization parameters for a similar task with limited data . In the For the omics data analysis of cancers, the transfer learning strategy has been applied to different tasks. Li et al. built a pan-cancer Cox model for the prediction of survival time, where 8 cancer types were combined to assist the training of a target cancer dataset . YousefiIn the present study, we propose a new method to use a transfer learning–based neural network for imputing gene expression from DNA methylation data, namely, TDimpute. Specifically, we first train a neural network based on the pan-cancer dataset to build a general imputation model for all cancers, which is then transferred to target cancer types Fig. . To the G is the raw gene expression value. For DNA methylation data, we excluded methylation sites with “NA” values, and 269,023 methylation sites remained. By further removing sites with small variances (<0.05) over all samples, 27,717 CpG sites were kept. Here, for evaluating all imputation methods we kept only samples having both RNA-seq and DNA methylation data. Finally, the dataset contains 8,856 samples with expression data for genes and methylation values of 33 cancers, namely, TCGA dataset. To keep enough sample size for downstream analyses, we selected cancer types containing >200 samples with complete DNA methylation, gene expression, and clinical data, leading to 16 cancer types for the test  that has the smallest sample size. Here, the DNA methylation data were downloaded from the TARGET Data Matrix , and theThe imputation methods were evaluated by the mean values of the root mean square error (RMSE), mean absolute error (MAE), and the squared Pearson correlation coefficient  were additionally compared with the prognosis-related gene list downloaded from The Human Protein Atlas (THPA) [The top 100 genes (ranked by s (THPA)  by the eThe PR-AUC, overlap of top 100 prognostic genes, and the enrichment factors per cancer dataset are detailed in Tables S3, S4.2, and S5, respectively.We also evaluated the effects of different imputation methods on clustering analysis and survival analysis. By input of the top 100 prognosis-related genes, the K-means algorithm was used to divide the samples into 2 clusters. Fig. A further survival analysis  in the survival curves according to the log-rank test. By comparison, the P-values were 1.1e−03, 1.9e−03, 9.5e−03, and 1.2e−03 for TDimpute-self, Lasso, TOBMI, and SVD, respectively.As a real data application, we performed survival analysis on the uterine corpus endometrial carcinoma (UCEC) dataset because it has the largest proportion of missing gene expression in TCGA (172 samples with RNA-seq and 267 without). By using the imputed samples, TDimpute achieved the highest C-index of 0.588, compared with TDimpute-self, SVD, TOBMI, and Lasso with C-index of 0.575, 0.55, 0.553, and 0.508, respectively. P-value of 7.81e−08, compared with 4.53e−07, 8.64e−06, 3.80e−03, and 2.99e−02 for TDimpute-self, TOBMI, Lasso, and SVD, respectively. TDimpute achieved the highest C-index of 0.592, compared with 0.591, 0.523, 0.558, and 0.501 for TDimpute-self, SVD, TOBMI, and Lasso, respectively.As an independent test beyond TCGA, we selected the smallest Wilms tumor dataset from the TARGET project. By fine-tuning the TCGA pan-cancer model with 59 randomly selected samples (50% of the dataset), the model was tested on the remaining samples. As expected, TDimpute achieved the lowest RMSE of 0.955 . This demonstrates the generalization of the TCGA-pretrained model on an independent dataset. The K-means method is used to cluster the 118 samples after imputation, and the 2 resulting clusters were used to plot the survival curves. For the Kaplan-Meier survival curves in Fig. In this study, we used TDimpute to perform missing gene expression imputation by building a highly nonlinear mapping from DNA methylation to gene expression data. Owing to the limited size of cancer datasets in TCGA, we used transfer learning to capture the commonalties in the pan-cancer dataset for pretraining parameters. We compared TDimpute with and without transfer learning, Lasso, SVD, and TOBMI methods by RMSE, MAE, and correlation Experimental results on 16 cancer datasets confirmed that our TDimpute method without transfer learning already outperformed the Lasso, SVD, and TOBMI methods in different evaluation metrics. By including transfer learning, the TDimpute method can further improve its performance especially at high rates of missing data. In addition, the ranking of the Lasso, SVD, and TOBMI methods on imputation accuracy . The network can be considered as a highly nonlinear regression function that maps DNA methylation data (input) to gene expression data (output).As shown in Fig. The network was trained using Adam optimizer with default parameters (learning rate set as 0.0001) . The metTo train the prediction model for 1 target cancer in TCGA, the datasets of other cancer types were combined to generate a multi-cancer model that was then fine-tuned by the target cancer data , we first optimized the pretrained model to determine the architecture of the neural network, i.e., the number of hidden layers, the hidden layer size, and the epochs to stop training. Details on the performance analysis for each hyper-parameter are provided in Finally, we selected the following hyper-parameters for the pan-cancer model: 1 hidden layer (selected from 1 and 3) including 4,000 nodes , Sigmoid activation function , epochs of 300 , and batch size of 128. For the fine-tuning stage, 150 epochs  were used and the batch size was set as 16 because of small sample sizes under large missing rates. Dropout was not used because it decreased the performance .In the transfer learning for each target cancer type of TCGA, 32 cancer types in the TCGA dataset (except the target cancer type) were used as the source domain dataset for pretraining a model. During the test, to simulate the performance under different missing rates, we used 5 fractions . At each missing rate, we used the sample function (without replacement) in Pandas  to get 1We compared our method with other imputation methods, including Lasso , TOBMI , and SVDHere, we used the squared Pearson correlation coefficient d) genes . Hence, P-value describing the significance of the relation between the gene and the target cancer, and genes with P-values A common task in the analysis of gene expression data is the identification of prognostic genes. To evaluate the effect of different imputation methods on the identification of potentially prognosis-related genes, we built univariate Cox proportional hazard regression models to select statistically significant genes correlated with overall survival. With the Cox model, each gene is assigned a In addition, we compared our identified genes with the list of prognosis-related genes from THPA  through P-value output from the univariate Cox models. By using the top 100 genes, their expression values were used to divide samples into 2 clusters by the K-means method. Clustering performance was assessed by means of the ARI, a measure of agreement between the predicted clustering labels (by imputed datasets) and the actual clustering labels . We further made survival predictions with significantly related genes : Platform independentProgramming language: PythonLicense: MITbiotoolsID: TDimputeRRID:SCR_018306GigaScience repository, GigaDB [The datasets and pretrained pan-cancer models supporting the results of this article are available in the Synapse with ID: syn21438134 . Snapsho, GigaDB .Figure S1. Mean absolute error of each imputation method. Results were averaged across 16 imputed cancer datasets. TDimpute-self indicates the TDimpute trained and predicted on the target cancer dataset. TDimpute-noTF indicates the TDimpute trained on the pan-cancer dataset (excluding the target cancer) and predicted on the target cancer dataset. The error bars show the standard deviation.Figure S2. RMSE on 16 imputed cancer datasets with different missing rates. The results were averaged over 5 random replicates. The error bars show the standard deviation.Figure S3. The squared Pearson correlation coefficients Figure S4. The squared Pearson correlation coefficients Figure S5. ARI on 16 imputed cancer datasets with different missing rates. The results were averaged over 5 random replicates. The error bars show the standard error of the mean.Figure S6. C-index on 16 imputed cancer datasets with different missing rates. The results were averaged over 5 random replicates. The error bars show the standard error of the mean.Figure S7. Kaplan-Meier plot for the 2 clusters obtained from the UCEC dataset imputed by TDimpute and SVD, respectively.Figure S8. The architecture of transfer learning–based neural network (TDimpute) with cancer-specific CpG sites as auxiliary input. The input at the fine-tuning stage consists of 2 parts: the cancer-specific part takes the highly variable CpG sites from the target cancer as input, and the transfer part takes the commonly variable CpG sites from the pan-cancer dataset as input.Figure S9. RMSE values of each imputation method with top 20,000 CpG sites as input. For SVD, TOBMI, Lasso, and TDimpute-self, the top 20,000 variable cancer-specific CpG sites are used as input, while the input of TDimpute includes both the pan-cancer and cancer-specific CpG sites. Results were averaged across 16 imputed cancer datasets. The error bars show the standard deviation.Figure S10. Loss curves for different hidden layer shape.Figure S11. The loss curves of different missing rates on the validation dataset of BRCA.Table S1. PR-AUC for detecting methylation-driving genes on imputed cancer datasets over 16 cancer types.Table S2.1. Overlap of top 100 methylation-driving genes from imputed dataset and full dataset.Table S2.2. Overlap of top 100 methylation-driving genes between imputed dataset and full dataset over 16 cancer types.Table S3. PR-AUC for detecting significantly prognostic gene on imputed datasets over 16 cancer types.Table S4.1. Overlap of top 100 significantly prognostic genes identified by univariate Cox model between imputed datasets and full datasets.Table S4.2. Overlap of top 100 prognostic genes identified by univariate Cox model between imputed dataset and full dataset over 16 cancer types.Table S5. The enrichment factors of the top 100 ranked genes in the gene list from The Human Protein Atlas across 16 cancer types.Table S6. Hyper-parameter analysis for hidden layer shape, the number of hidden layers, and training epochs on pan-cancer dataset (excluding BRCA dataset).giaa076_GIGA-D-19-00438_Original_SubmissionClick here for additional data file.giaa076_GIGA-D-19-00438_Revision_1Click here for additional data file.giaa076_GIGA-D-19-00438_Revision_2Click here for additional data file.giaa076_Response_to_Reviewer_Comments_Original_SubmissionClick here for additional data file.giaa076_Response_to_Reviewer_Comments_Revision_1Click here for additional data file.giaa076_Reviewer_1_Report_Original_SubmissionAli Sharifi-Zarchi -- 2/18/2020 ReviewedClick here for additional data file.giaa076_Reviewer_2_Report_Original_SubmissionJiangning Song -- 2/23/2020 ReviewedClick here for additional data file.giaa076_Reviewer_2_Report_Revision_1Jiangning Song -- 5/13/2020 ReviewedClick here for additional data file.giaa076_Supplemental_FileClick here for additional data file.ARI: adjusted Rand index; C-index: concordance index; CNN: convolutional neural network; DAE: denoising autoencoder; FDR: false discovery rate; KNN: k-nearest neighbor; LASSO: least absolute shrinkage and selection operator; MAE: mean absolute error; PR-AUC: area under precision-recall curve; RMSE: root mean square error; SVD: singular value decomposition imputation; TARGET: Therapeutically Applicable Research to Generate Effective Treatments; TCGA: The Cancer Genome Atlas; TDimpute: transfer learning–based deep neural network for imputation; THPA: The Human Protein Atlas; TNM: tumor–node–metastasis; TOBMI: trans-omics block missing data imputation; UCSC: University of California Santa Cruz.The authors declare that they have no competing interests.This work has been supported by the National Key R&D Program of China (2018YFC0910500), National Natural Science Foundation of China , Guangdong Key Field R&D Plan (2018B010109006 and 2019B020228001), Natural Science Foundation of Guangdong, China (2019A1515012207), and Introducing Innovative and Entrepreneurial Teams (2016ZT06D211).Y.Y. and C.L. conceived the study. Y.Y. and X.Z. contributed to the design and development of the model. X.Z. and H.C. implemented the software and led the curation of the datasets. Y.Y., H.Z., and X.Z. led the analytics. All authors wrote and approved the final manuscript.


In early breast cancer (BC), five conventional biomarkers—estrogen receptor (ER), progesterone receptor (PgR), human epidermal growth factor receptor 2 (HER2), Ki67, and Nottingham histologic grade (NHG)—are used to determine prognosis and treatment. We aimed to develop classifiers for these biomarkers that were based on tumor mRNA sequencing (RNA-seq), compare classification performance, and test whether such predictors could add value for risk stratification.In total, 3,678 patients with BC were studied. For 405 tumors, a comprehensive multi-rater histopathologic evaluation was performed. Using RNA-seq data, single-gene classifiers and multigene classifiers (MGCs) were trained on consensus histopathology labels. Trained classifiers were tested on a prospective population-based series of 3,273 BCs that included a median follow-up of 52 months , and results were evaluated by agreement statistics and Kaplan-Meier and Cox survival analyses.Pathologist concordance was high for ER, PgR, and HER2  but moderate for Ki67 and NHG . Concordance between RNA-seq classifiers and histopathology for the independent cohort of 3,273 was similar to interpathologist concordance. Patients with discordant classifications, predicted as hormone responsive by histopathology but non–hormone responsive by MGC, had significantly inferior overall survival compared with patients who had concordant results. This extended to patients who received no adjuvant therapy , or endocrine therapy alone . For cases identified as hormone responsive by histopathology and who received endocrine therapy alone, the MGC hormone-responsive classifier remained significant after multivariable adjustment .Classification error rates for RNA-seq–based classifiers for the five key BC biomarkers generally were equivalent to conventional histopathology. However, RNA-seq classifiers provided added clinical value in particular for tumors determined by histopathology to be hormone responsive but by RNA-seq to be hormone insensitive. The NHG system was developed to establish better standards and improve reproducibility, and it is the recommended method for BC grading today. NHG reproducibility studies12 have reported modest agreements , which correspond to 15% to 30% discordance.Current evaluation of these BC biomarkers is imperfect. Immunohistochemistry (IHC) is the principal method for ER, PgR, HER2, and Ki67 measurement, and in situ hybridization (ISH) methods are used to refine HER2 IHC. Among laboratories, significant differences exist in, for example, fixation, antigen retrieval, antibodies, chemistries, scoring systems, and interpretation. Accuracy and reproducibility are concerns, with up to 20% false-positive or false-negative ER/PgR IHC determinations.1316 Massively parallel sequencing of mRNA (RNA-seq) has advantages compared with earlier methods, including greater dynamic range and reproducibility and the ability to discover and quantify transcripts without a priori sequence knowledge. In 2010, toward implementation of molecular profiling in the clinical routine, we launched the Sweden Cancerome Analysis Network Breast Initiative , an ongoing population-based multicenter observational study covering a wide geography of Sweden that prospectively invites all patients with BC to participate.17 To date, approximately 85% of the eligible catchment population are included, more than 11,000 patients have enrolled, and blood and fresh tumor tissues are sampled for molecular research. In the first phase, all tumors are analyzed by RNA-seq generally within 1 week after surgery. Thus, for each BC, it will be possible to report a multitude of biomarker tests simultaneously on the basis of its RNA-sequencing data and within a clinically actionable time frame.Microarray and reverse transcriptase polymerase chain reaction–based gene expression analyses of BCs have yielded many signatures for tumor subtyping, prognosis, and survival, as well as for individual biomarkers, such as ER, PgR, HER2, and PTEN.Herein, we aimed to validate the SCAN-B multicenter infrastructure and provide molecular analyses of clinical value by developing RNA-seq–derived classifiers for the conventional histopathologic BC biomarkers ER, PgR, HER2, Ki67, and NHG. For this purpose, both single-gene classifiers (SGCs) and multigene classifiers (MGCs) were developed by using a training cohort, the prediction accuracy was compared against current clinical practice across a large independent prospective cohort, and the classifier predictions and their discrepancies to histopathology were evaluated with respect to patient survival.17 . The diagnostic slides and newly stained slides were each scored in total by three pathologists independently by using 1% or greater tumor cell staining threshold for hormone receptor positivity, standard HER2 HercepTest  and ISH criteria , greater than 20% positive nuclei for Ki67-high status, and the NHG scoring system (Data Supplement). On the basis of all evaluations, a consensus score for each biomarker was determined with the majority scores.18 All data are available from the NCBI Gene Expression Omnibus (Accession Nos. GSE81538 and GSE96058).Snap-frozen (training cohort) or RNAlater-preserved  tumor specimens were processed and sequenced, and the raw data (Data Supplement) was processed as described previously.ESR1, PGR, ERBB2, and MKI67 that maximized concordance to the respective histopathology consensus score (Data Supplement). MGCs for ER, PgR, HER2, Ki67, and NHG were built by training nearest shrunken centroid (NSC)19 models with the 5,000 most varying genes across the training cohort (Data Supplement) and the histopathology consensus scores as training labels. Within the training set, optimal model parameters were determined by using cross-validation and then were used to train prediction models with all training samples. The resulting four SGCs and five MGCs were used to predict the biomarker status of 3,273 independent validation BC samples. The biologic functional annotation clusters of each MGC signature were evaluated with the DAVID Bioinformatics Resource.20Within the 405-patient training set, SGCs were built for the ER, PgR, HER2, and Ki67 biomarkers by determining the optimal expression thresholds for the genes 21 (defined in the Data Supplement) and balanced statistics—Cohen’s κ and Matthews correlation coefficient (MCC)—and were interpreted according to Viera and Garrett.22 The κ and MCC values were comparable (Data Supplement), so we focused on κ. Kaplan-Meier and Cox regression survival analyses were performed with overall survival as the end point. Multivariable Cox models included the variables age at diagnosis, lymph node status, tumor size, ER, PgR, HER2, and NHG as covariates, as relevant (Data Supplement). All calculations were performed with R 3.2.3. P values of ≤ .05 were considered significant.Histopathology evaluations and single-gene and multigene predictions were compared with agreement statisticsTo estimate the inherent variability within clinical histopathology and to determine a consensus score for each BC biomarker for classifier training, a comprehensive histopathologic analysis was performed for 405 patient breast tumors with three readings of up to two independent stains for the five conventional biomarkers: ER, PgR, HER2, Ki67, and NHG . With thESR1 (which encodes the ER protein), PGR (PgR), ERBB2 (HER2), and MKI67 (Ki67) (Data Supplement). Next, MGCs were trained, and the training-cohort cross-validation accuracy was determined  as follows: ER, 95.3% ± 2.4%; PgR, 90.4% ± 2.9%; HER2, 88.5% ± 3.8%; Ki67, 84.9% ± 3.4%; and NHG, 73.8% ± 3.9%. For MGCs, the NSC method has the property of eliminating noninformative genes (zero weight for the classification). The ER classifier had 459 weighted genes; PgR, 184; HER2, 312; Ki67, 273; and NHG, 206 (Data Supplement). In total, 869 genes had nonzero weights in at least one MGC classifier. The constituent biologic themes for each MGC classifier were investigated with functional annotation clustering (Data Supplement).Whole-transcriptome expression profiles were generated for the 405 training samples using RNA-seq. For the SGCs, optimal thresholds were determined for To evaluate the classifiers, we tested them on RNA-seq data generated for 3,273 independent tumors from the prospective population-based multicenter SCAN-B study . Concordance between the diagnostic histopathologic results and the SGC predictions was substantial for ER  and HER2  and moderate for PgR  and Ki67  as well as within subgroups defined by adjuvant treatment: no systemic therapy  and only endocrine therapy . Ultimately, these results can be used to identify patients who may benefit from additional treatment. Another approach is to use clinical outcome as the training labels to develop new prognostic/predictive signatures.34 The SCAN-B material is excellently suited to evaluate previously published signatures; as we accrue longer follow-up, we aim to develop RNA-seq signatures trained on clinical outcomes.Despite these possible explanations for discrepancies, when benchmarked against patient outcome, our classifiers exemplified enhanced stratification of patients with significant differences in overall survival  and A5. 11 To our knowledge, this study is the first to develop a validated predictor for Ki67 status. The lower concordance between our Ki67 predictions compared with the clinical reference is related to the relatively larger Ki67 interrater disagreement seen within our consensus pathology evaluation, which is likely a consequence of the continuous nature of Ki67 expression and of the spectrum of proliferation activity and pathways in BC.Ki67 has been introduced relatively recently in international guidelines.12 Most misclassified tumor cases in this study were histologically grade 1 (G1) or grade 3 that were misclassified as grade 2 (G2) by our predictor. Large interrater disagreement, especially for G1 and G2, could explain the results of our classifier with only moderate OA to histopathology (67.7%). All histologic G1 occurrences were misclassified, which may have been a result of the imbalanced composition of the training set for NHG (48 of 405 samples consensus-scored G1), or may have occurred because G1 is not a discrete entity but rather the lower end of an underlying continuous scale. Indeed, Kaplan-Meier analysis showed that the curves G1 and G2 largely overlapped in the validation cohort (Appendix 36 and essentially is what our NHG predictor has become.NHG is distinct from the other biomarkers. It has no single underlying gene but rather is a compound biomarker that consists of three morphologic properties: tubular differentiation, nuclear pleomorphism, and mitotic count. Moreover, NHG prediction is a three-class problem. Even for pathologists, NHG can be difficult to determine, as evidenced by the moderate κ and OA results within clinical pathology, in line with the literature.Appendix . AnotherAn important question when building classifiers is how many genes to use. We compared single-gene and multigene classifiers. When compared with clinical pathology, SGCs have slightly better concordance than MGCs for ER and HER2, whereas the SGC and MGC performances were comparable for PgR and Ki67. This difference may have developed because these biomarkers are faithfully represented by their associated single genes. Another consideration for classifiers is robustness toward missing values. MGCs may be more robust than SGCs, because they are able to classify tumors correctly even when the main gene that underlies a biomarker is poorly measured in a particular analysis. When clinical outcome was considered, the survival analyses indicated that our MGCs generally contained greater potential clinical utility than SGCs to complement histopathology.In summary, we have performed a systematic pathologic evaluation of 405 BC tumors, which resulted in consensus scores for the five conventional BC biomarkers and estimated a well-controlled best-case scenario for the inherent uncertainty within clinical histopathology. With tumor RNA-seq data and the consensus scores, we trained SGCs and MGCs and evaluated the classifiers on an independent set of 3,273 tumors. The accuracy of our classifiers was comparable to the inherent accuracy of clinical pathology and was highly reproducible. Classifiers based on the expression of single genes performed slightly better than MGCs for concordance to histopathology, but MGCs performed significantly better for stratification of patients into groups with clinically meaningful differences in survival, in particular for histopathologic hormone-responsive BCs. In conclusion, RNA-seq–based classifiers may be suitable complementary diagnostics for BC, in particular for difficult diagnoses in which the classifier can add an additional vote toward the therapeutic choice. For future implementation of our MGCs in the clinical routine, additional health economics analyses and external validation are needed.


The “sigFeature” R package is centered around a function called “sigFeature,” which provides automatic selection of features for the binary classification. Using six publicly available microarray data sets (downloaded from Gene Expression Omnibus) with different biological attributes, we further compared the performance of “sigFeature” to three other feature selection algorithms. A small number of selected features (by “sigFeature”) also show higher classification accuracy. For further downstream evaluation of its biological signature, we conducted gene set enrichment analysis with the selected features (genes) from “sigFeature” and compared it with the outputs of other algorithms. We observed that “sigFeature” is able to predict the signature of four out of six microarray data sets accurately, whereas the other algorithms predict less data set signatures. Thus, “sigFeature” is considerably better than related algorithms in discovering differentially significant features from microarray data sets.Biological data are accumulating at a faster rate, but interpreting them still remains a problem. Classifying biological data into distinct groups is the first step in understanding them. Data classification in response to a certain treatment is an extremely important aspect for differentially expressed genes in making present/absent calls. Many feature selection algorithms have been developed including the support vector machine recursive feature elimination procedure (SVM-RFE) and its variants. Support vector machine RFEs are greedy methods that attempt to find superlative possible combinations leading to binary classification, which may not be biologically significant. To overcome this limitation of SVM-RFE, we propose a novel feature selection algorithm, termed as “sigFeature” ( Feature selection is described as a process in which a subset of relevant features is selected from a larger data set. These features are used for model construction. On the basis of the model created, one can separate or classify the classes present in a data set  on microarray data sets to obtain the statistically significant differentially expressed genes (DEGs). However, the DEGs produced by most of the methods often contain biologically irrelevant data lacking discriminatory power to classify the groups present in the data set. This is also the same with the clinical data. It is therefore essential to develop an appropriate algorithm to identify biologically significant DEGs together with excellent classification precision in a tumor versus normal tissue contrast  feature selection algorithms focus mostly on classification accuracy of the selected features, which often fails to be biologically relevant leading to inaccurate downstream data analysis  seamlessly. After comparing “sigFeature” with three selection algorithms such as “SVM-RFE,” “SVM-T-RFE,” and “SVM-BT-RFE”  and -BT-RFE”  using th-BT-RFE”  was doneWe have chosen six publicly available microarray data sets randomly for testing and comparison with our tool. All the feature selection algorithms are applied precisely for cancer classification. In addition to most feature selection methods embedded with an estimation of the classifier, “sigFeature” is inherently a multivariate method. “sigFeature” evaluates the relevance of several features considered together. A univariate method, on the other hand, assesses the significance of each feature individually. The latter is often computationally easier, but the former is more sophisticated from the data analysis point of view, as genes are known to communicate in many respects and are often coregulated. We also used the selection approach of the ensemble feature that relies on various subsamples of the original data to create different signatures. Using GSEA analysis, the selected robust signatures are finally analyzed to determine its biological signature.All these data sets are preprocessed using the following procedures.The purpose of data normalization is to minimize the variation in data due to various non-biological factors and make them comparable in one scale. These data sets are normalized by “quantile” normalization method using the “Bioconductor” package “Limma” ]. For feature selection, we have used SVM in “sigFeature,” with some modification to obtain the desired weight values for each feature. In this section, we have explained how SVM effectively operates, because our algorithm is built upon the basic principle of SVM.x1, y1), …, , yi ∈ }, where x =  is a d-dimensional input vector, and y is the class label, and w =  are coefficients of the hyperplane, and b represents the intercept of the hyperplane. Support vector machines are used in different feature selection algorithms, for example, “SVM-RFE,” “SVM-T-RFE,” and “SVM-BT-RFE.” A detailed description of those algorithms is given in the Support vector machine uses kernel functions to perform classification efficiently on non-linear data. It implements implicit mapping in the high-dimensional input feature vectors into high-dimensional spaces. This produces a linear hyperplane, which can separate two groups of data meaningfully Butte, . The hypith feature.In 2002, Guyon et al.  and denoting the difference of the jth feature of the two classes such as jth element of w (weight vector) is considered, as wj and d is the total number of features. In special cases, where one class has only one sample, and the other class has more than one sample, t, the significant differences between the two classes can be calculated by using standard single sample t statistic where jth feature, and n+ and n− are the number of samples present in each class (+1 class and −1 class), respectively.Generally, the expression data set contains a large number of probes sets and a small number of sample sizes. It is observed that the number of samples in each class is unequal in the data set. In our approach, the features to be selected for classification need to contain maximum discriminatory power between the classes. Considering the imbalance in sample size (no. of samples class 1 ≠ class 2) into consideration, we characterize the accompanying measure as below (Equation 2):Ho and move on in the direction of a specified alternative hypothesis Ha.v is the level of flexibility parameter for the corresponding reference distribution, and u is the observed (positive) estimation of the test statistic on the basis of the level of flexibility v. Unlike “SVM-RFE,” in “sigFeature” we select important features by incorporating the product of weights and the corresponding differences between the classes. The pseudo code of the algorithm is given as below.We have further embedded the recent concept of ensemble feature selection techniques to improve the stability of feature selection algorithms  in a final signature. A general formula obtained by summing the ranks over all bootstrap samples for the ensemble ranking f is as follows:wi denotes a bootstrap-dependent weight.We chose our training set with 40 subsampling sets consisting of 90% of the original data set. The remaining 10% of the data can be used to evaluate classification performance as an independent validation set. The main motto behind this is to generate a variety of feature selection protocols using “sigFeature” algorithm. Because the “sigFeature” algorithm is deterministic, performing it on different training samples is the only way to generate diversity in selection. To this end, we use the bootstrapping method, a well-established statistical technique for reducing variance Efron, . In statf is obtained by simply summing the ranks above all samples of the bootstrap.Then, the ensemble ranking s features with the highest summed rank are selected from f to select the final set of features for a size s signature.The Stability measurement is an essential part of the classification performance. It is therefore necessary to incorporate stability, else any feature selection algorithm may always return the same features regardless of any input sample combinations. We used Kuncheva index (KI) statistical analysis described below to measure stability of the selected features.X = {x1, …., xM} with M samples and N features. After that, the sample set is subsampled randomly with k times of size [xM](0 < x < 1). We randomly selected k = 40 in our data analysis and x = 0.9. Consequently, on each of the k subsamplings, feature selection is performed, and a marker set of a given size is selected, further referred to as a signature. The relative stability Stot can then be defined as the average of all pair-like comparisons between all the signatures on the k subsamplings.fi is the signature obtained by the subsampling method i(1 ≤ i ≤ k). KI denotes the KI. A stability index defined as follows between fi and fj  ≤ 1, and the higher its value, the greater the number of features commonly selected in both signatures. In this index, the s2/N term corrects a bias due to the chance to select common features between two randomly selected signatures.Let us consider a data set In 2002, Ambroise and McLachlan  to compare selected features using the aforementioned algorithms for further downstream evaluation. MSigDB V 7.0 is a compilation of annotated gene sets. For this analysis, Cancer Gene Neighborhoods (CGN), Cancer modules (CM), and Oncogenic signatures (C6) gene sets were used during GSEA . We use SVM by using a CRAN package named “e1071”  are implemented in R script. The “sigFeature” package is available in Bioconductor and GitHub repository  for reestablishing the robustness of the newly developed algorithm , which are produced by the different algorithms. Although individual feature lists are produced by each subset (40 bootstraps), we average the p-value by its position in the feature list. In each data set, it may not create an equal number of sample size in both classes. We use default t-test (two-sided) using an R function named t-testand computed the unadjusted p-values. Then the unadjusted p-values produced are used to plot a histogram, depicted in p-values associated with the rank feature. The histogram plot shows the area proportional to the unadjusted p-value frequency.In order to obtain a meaningful biological insight, the selected features must have a significant difference between the sample groups present in the data sets. In this experiment, we compare the characteristics (significant difference between the groups) of the features produced by “sigFeature” with other algorithms. Thus, we compare the The features identified by newly developed “sigFeature” (top five hundred features by CLA) and other three algorithms SVM-RFE, SVM-T-RFE, and SVM-BT-RFE were analyzed by the aforementioned workflow to check whether they are able to predict the biological attribute of the six microarray data sets from which they were derived.p ≤ 0.05) predict two gene sets “Neighborhood of RAP1A” and “Neighborhood of UBE2N.” Both genes were strongly involved in regulation of tumor metastasis. The monomeric G protein, RAP1A, acts as a switch during transduction of cellular signaling and generally regulated by its binding to either guanosine triphosphate (GTP) or guanosine diphosphate. It functions to regulate the function of cell adhesions and junction proteins and mediate cellular migration and polarization to promote metastasis in prostrate tumors, ovarian tumors, melanoma, lung cancer, glioma, bladder cancer, leukemia, and also oral cavity  “Module_5 Lung genes.”Data set GSE26712 is based on ovarian cancer. Feature selected by “sigFeature” also predicted its biological origin with the enrichment of oncogenic signature (C6) “Module_1 Ovary genes.” Among the other algorithms, only SVM-RFE–derived feature was able to predict that the data set is of ovarian origin by enrichment of the same module.Data set GSE3744 is composed of breast carcinoma samples. Only SVM-BT-RFE was able to partially predict the true biological nature of the data set. The feature enriched the term oncogenic signature (C6) “KRAS.600.LUNG.BREAST_UP.V1_UP” that comprised the genes that were up-regulated in epithelial lung and breast cancer cell lines overexpressing an oncogenic form of KRAS gene. The “sigFeature” and other given algorithms could not predict the biological nature of the data set. We needed further in-depth inquiry to define the appropriate reason why those algorithms were unable to detect the cancer signature.p < 0.05 and a fold change of ±1.5. This list of signatures was also unable to predict the cancer signature in the GSEA assessment.Next, data set GSE25099 has the characteristic of oral squamous cell carcinoma. In this case, “sigFeature” and the other algorithms could not identify the biological signature, as they could not enrich any gene set containing features related to oral cancer. For further clarification, a differential expression analysis was performed between case and control samples, and a signature list was generated using a cutoff value of Overall, the analysis revealed that “sigFeature” is able to predict the signature of three out of six data sets with absolute precision, whereas the fourth one partially . Comparat-test with robust list of features produced in each algorithm of selection. The p-values are graphically displayed below in p-value generated by the top characteristics  selected by the “sigFeature” algorithm is much more significant than the unadjusted p-value generated by the other selection algorithm. Also for the robust feature lists, the similar result is found.Next, we have tested the same t statistic  to select significant features. We also tested the “sigFeature” algorithm on six publicly available microarray data sets, containing different biological attributes, and compared it with already existing similar type of algorithms . The plots show that the sigFeature algorithm's initial goal of selecting significant feature along with excellent classification accuracy is being met. The top features chosen by the sigFeature algorithm are very considerably distinct from those chosen by other feature selection algorithms used in this research work. Thus, the average p-value plot indicates significant results for the features chosen by our algorithm. We compared the p-value produced by the top 1,000 features using those algorithms; it shows that more significant p-value s are generated by the features that are selected by “sigFeature” algorithm than others.In this article, we proposed a novel feature selection algorithm using “SVM-RFE” and We could also predict robust biomarkers with a concrete focus on microarray studies on six cancer diagnosis data sets using sigFeature algorithm. The stability of the markers is appropriate for both reproducibility and biological validation. However, stability alone is not a nice performance measure, as it is easy to improve stability by considering some fixed sets of features. Moreover, the ensuing predictive model is probably poor in the classification of new samples. The CLA technique is an experimental methodology for evaluating the strength of biomarker lists in combination with the predictive results of classification designs based on them. This CLA protocol repeatedly considers some samples for selecting markers as well as estimating classifiers from autonomous samples used to assess the efficiency of classification. The external 10-fold cross-validation error is more convenient to evaluate the predictive performance of data sets with unbalanced class proportions, a common situation for microarray experiments. “sigFeature” shows promising performance compared to other feature selection algorithms.Finally, performing GSEA analysis using specified cancer gene sets of MSigDB, we obtained a concrete evidence to show that features selected by “sigFeature” have the potential to identify biological attributes of data sets more accurately. Here, “sigFeature” was able to predict the signature of three out of six data sets with complete accuracy, whereas the fourth one partially. Comparable expertise was observed with SVM-RFE, whereas the rest have put up a comparatively poor show.Thus, we can conclude that the proposed algorithm identifies features leading to more accurate classification and generation of differentially significant features.Publicly available datasets were analyzed in this study. This data can be found here: GSE3744, GSE25099, GSE26712, GSE2280, GSE7670, and GSE4290.Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The ethics committee waived the requirement of written informed consent for participation.PD developed the method and implemented the sigFeature algorithms, under the supervision of ST and SR. All authors  added to composing the content, read and affirmed the final version of the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Single-cell RNA-seq (scRNA-seq) profiles gene expression of individual cells. Unique molecular identifiers (UMIs) remove duplicates in read counts resulting from polymerase chain reaction, a major source of noise. For scRNA-seq data lacking UMIs, we propose quasi-UMIs: quantile normalization of read counts to a compound Poisson distribution empirically derived from UMI datasets. When applied to ground-truth datasets having both reads and UMIs, quasi-UMI normalization has higher accuracy than competing methods. Using quasi-UMIs enables methods designed specifically for UMI data to be applied to non-UMI scRNA-seq datasets. Single-cell RNA-seq (scRNA-seq) has become a standard tool for measuring gene expression patterns from individual cells. The initial molecule capture and reverse transcription (RT) steps in scRNA-seq protocols result in low quantities of cDNA, so a large number of PCR cycles are needed to produce enough material for measurement. The resulting libraries, that are then sequenced, contain many duplicates of each of the single mRNA molecules extracted from the original cell . To accoIn both UMI and read count data, the fraction of zeros per cell is often a dominant source of variation. Not only does the zero fraction strongly correlate with the first principal component, but it also affects the entire gene expression distribution . While tcensus counts method transforms read counts and attempts to match the underlying UMI distribution based on the key observation that the mode of the nonzero UMI count distribution is typically one [The substantial distortions in read counts have motivated the development of sophisticated normalization procedures. One approach is to attempt to transform the data to more closely follow a normal (Gaussian) noise model. For example, log-transformed expression values after normalization by transcripts per million (TPM), scran , or SCnoally one . Rather ally one , 12. Centarget distribution. The most widely implemented version of quantile normalization generates the target distribution by averaging over empirical distributions from the data [Here, we present quasi-UMIs (QUMIs), a normalization technique for scRNA-seq read counts that, like census counts, attempts to match the UMI count distribution. Our approach differs from census counts in that we apply quantile normalization rather than a linear transformation, producing a discrete distribution. In general, quantile normalization forces all cells to follow a specific the data . In the the data . We therthe data , 15. Thithe data , TPMs caWe used thirteen public scRNA-seq datasets Table . For theWe explored the effects of normalization on read counts from both UMI and non-UMI protocols. The variability introduced by PCR resulted in hundreds of genes with read counts above 100 that mapped back to less than five UMI counts, in some cases just one Fig. a, d–f. DTo quantile normalize read counts to match UMI counts, we identified the qualitative characteristics of the target UMI count distribution. Due to the heavy tail of UMI counts, log-log plots  are an eA recent survey of a wide variety of datasets found that ostensible power law relationships are better described by lognormal distributions . We therWe first sought to quantify goodness of fit by computing the Bayesian information criteria (BIC)  for eachAssuming the underlying UMI count distribution is Poisson-lognormal, only two parameters are needed to describe each cell: scale and shape. If UMI data are available, these are easily estimated using maximum likelihood (MLEs). However, in read count data without UMIs, this is not possible due to PCR distortion. Conveniently, if the shape parameter is known a priori, the scale parameter can be estimated from the fraction of zeros. This is useful because the zero fraction derived from read counts equals the zero fraction in the UMI counts for the same cell; zero is the only expression value in read count data that is not altered by PCR bias. Therefore, if the shape parameter is assumed known, the target distribution for a given cell can be determined from the read count data. Our method requires the shape parameter to be fixed. To determine reasonable shape parameter values, we computed MLEs for all cells in the training data.The shape parameters varied both within and between training datasets, with values ranging from 1.0 to 3.0 Fig. . We therWhile it was necessary to fix the shape parameter before applying quantile normalization to the test data, each cell was allowed to have its own scale parameter. If the scale parameters were also held fixed, the QUMI target distribution would be identical for every cell and would predict a constant zero fraction across cells. But this is discordant with the fact that UMI count data exhibit variation in the zero fraction across cells . Since tAfter estimating the scale parameter for each cell, we obtained empirical quantiles (ranks) from read counts and transformed the ranks to QUMI counts by matching to the target distribution’s theoretical quantiles  counts with census counts and unnormalized read counts. We quantified the accuracy of a normalization method for a given cell by computing the Euclidean distance between the log of the normalized count vector and the log of the UMI count vector. Zero values were omitted from the computation because all of the normalization methods preserved the sparsity structure of the read counts.Across datasets, QUMI counts had the highest accuracy , while census counts were more accurate than read counts Fig. . The impThe accuracy of QUMI counts was not strongly affected by the choice between default and custom shape parameters. This could be due to the custom parameters being close to the default value for these particular test datasets  analysis, we obtained read counts and UMI counts from the Vieira Braga dataset Table , a dropsQuasi-UMI counts may be analyzed as if they were UMI counts. To illustrate this, we applied quasi-UMI normalization, scran , and cenWe examined the effects of normalization on downstream dimension reduction using principal component analysis (PCA) , GLM-PCAWe examined the ability of QUMI counts to profile a heterogeneous tissue using the Segerstolpe pancreas dataset . The oriWe have shown that UMI counts can be approximated by quantile normalization of read counts to quasi-UMIs (QUMIs) in scRNA-seq. The Poisson-lognormal model fits UMI count data well and can be used as a target distribution for QUMI normalization. However, the conceptual framework is generalizable to any discrete distribution that can be calibrated against UMI data, such as the Poisson-Lomax or two-component mixture models of active and inactive genes . Using tQUMI normalization mitigates the distortion of PCR amplification in scRNA-seq protocols that lack UMIs while preserving sparsity. However, just like the use of proper UMIs, it does not normalize differences between cells resulting from variation in efficiency of capture or reverse transcription. This contributes to differences in the zero fraction across cells, which are intentionally preserved in QUMI normalization. These sources of technical variation should be addressed through UMI-specific count models such as GLM-PCA or its approximations using residuals , 12.QUMI counts do not directly account for PCR bias arising from differences in gene length or GC content. These biases are not specific to single-cell protocols and have been addressed in the bulk RNA-seq literature . Since QA major advantage of QUMI counts is that they can be analyzed as if they were UMI counts. This avoids the need to develop customized methods of dimension reduction and feature selection for the read count distribution. Here, we have focused specifically on scRNA-seq read counts, but traditional bulk RNA-seq read count data is also affected by distortion from PCR amplification. While it may be possible to extend the QUMI framework to bulk RNA-seq data, an appropriate target distribution would need to be identified. This is challenging because a bulk RNA-seq sample, unlike scRNA-seq, is typically a mixture of cell types with unknown proportions. Such a mixture is unlikely to be easily characterized by a simple two-parameter distribution. PCR distortion is also present in read counts from metagenomics experiments , 44.Finally, we caution that QUMIs are not substitutes for proper UMIs. If the latter can be used in an experiment, they will certainly be more effective than QUMIs in removing PCR distortions. QUMI normalization relies on assumptions, such as a fixed shape parameter, which may not be met in certain datasets. Indeed, we observed in the Klein dataset that the shape parameter was not constant across experimental conditions. However, based on our sensitivity analysis, the accuracy of QUMI counts was robust to misspecification of the shape parameter.http://atlas.gs.washington.edu/worm-rna/docs/. The Clark dataset [https://github.com/gofflab/developing_mouse_retina_scRNASeq. The Grun dataset [http://imlspenticton.uzh.ch:3838/conquer/. A preprocessed version of the Klein dataset [https://hemberg-lab.github.io/scRNA.seq.datasets/mouse/esc/. The Schiebinger dataset [http://linnarssonlab.org/cortex/, and low-quality cells were removed according to the same criteria used in the original publication. The Zhang dataset [The Cao dataset  was obta dataset  was obta dataset  was down dataset  http://i dataset  was down dataset  was down dataset  was down dataset  was downhttps://jdblischak.github.io/singleCellSeq/analysis/compare-reads-v-molecules.html. The Zheng dataset [https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/cd14_monocytes. The Vieira Braga dataset [The Macosko dataset  was obta dataset  was obta dataset  was obtahttps://github.com/willtownes/patel2014gliohuman. The Segerstolpe dataset [The Patel dataset  was down dataset  was obtaScran normalization was applied using version 1.14.6 of the Bioconductor R package. Census counts were obtained using version 2.14.0 of the monocle Bioconductor R package.The probability mass function (PMF) of a compound Poisson distribution is obtained by placing a prior on the rate parameter of an ordinary Poisson distribution:σ and scale (logarithmic mean) μ, the prior is a lognormal distribution with the same parameters:For the Poisson-lognormal distribution with shape (logarithmic standard deviation) α and scale θ, the prior is a Lomax (shifted Pareto) distribution with the same parameters:For the Poisson-Lomax distribution with shape (power law tail index) m and v represent the mean and variance of a given prior distribution in a compound Poisson model. The marginal mean of the compound Poisson is also m, and the marginal variance is m+v. For example, for the Poisson-lognormal, the mean is exp(μ+σ2/2) and the variance is m+(exp(σ2)−1)m2. The Poisson-Lomax distribution has such a heavy tail that its moments are only finite in certain parameter regions. If α>1, then the mean is θ/(α−1). If α>2, then the variance is Let For Poisson-lognormal, we evaluated the PMF using the R package sads. For Poisson-Lomax, we evaluated the PMF by using 1000 numerical quadrature points. For each cell in the training datasets, we obtained maximum likelihood estimates (MLEs) of compound Poisson model parameters  by numerical optimization using the R function optim. The median of the shape parameter distribution across cells was then used to calibrate the quasi-UMI target distribution in the test and prediction datasets.For each cell, we simulated a vector of gene expression using the fitted MLE parameters. We then identified the maximum expression value (count) for each simulated cell. The test statistic was defined as the log of the ratio of the simulated maximum divided by the observed maximum in the original UMI counts. Each cell then had its own test statistic. If the statistic was close to zero, that indicated the fitted model was well calibrated to the tail of the UMI count data. We therefore computed histograms of the test statistic’s distribution across all cells in each training dataset and compared how close the distribution was to a target of zero.f be the Poisson-lognormal probability mass function (PMF) with fixed shape parameter σ and unknown cell-specific scale parameter μi. For a given cell i, the theoretical probability of a zero is f (a function of μi only since σ is fixed). The empirical probability of zero is simply the fraction of genes with zero read counts in that cell, which we denote with μi is obtained by finding a root of the function μi.For each cell in the test data, we obtained a target quasi-UMI distribution by estimating the cell-specific scale parameter from the empirical zero fraction in read counts using the method of moments (MOM). Specifically, let Once a target distribution for an individual cell was determined, we computed the log of the theoretical CDF by cumulatively applying the log-sum-exp transformation to the log-PMF function, which provided numerical stability. We then renormalized the probability distribution to exclude the zero value since zero values in read counts result from UMI counts of zero and do not need to be adjusted. This resulted in a table with positive integer indices providing the quasi-UMI count value and corresponding zero-truncated CDF values indicating the probability of a random variable with the target distribution falling below that value, conditional on it being nonzero. We then converted the vector of read counts (or TPMs for Smart-seq2 data) from all genes in the cell to empirical quantiles (ranks). Each gene was then aligned to a CDF bin based on its rank. For example, if the zero-truncated CDF had values of 0.8 at 1 and 0.9 at 2, the first 80% of genes with lowest nonzero read count values would be assigned QUMI value of 1 and the next lowest 10% of genes would be assigned QUMI value of 2. Typically, a single gene was placed into the highest QUMI bin due to the heavy tail of the target distribution.p values using Fisher’s exact test for each gene for each normalized count matrix as well as the unnormalized read counts. We used the p values computed from the UMI counts as a ground truth and computed three distance metrics for each of the other normalizations using the R package amap. First, we computed the Manhattan distance between the vector of p values. Next, we computed the Kendall distance which is based on ranks rather than numeric values of the p values. Finally, we used the Holm method [p values for multiple comparisons and identified sets of differentially expressed genes at significance <0.05 for each normalization method. We then computed the Jaccard distance between these sets and the set identified using UMI counts.For the Vieira Braga dropseq dataset Table , we normm method  to adjusSince neither QUMI nor census normalization of TPM values removes cell-to-cell variation in total counts, we divided the normalized counts by the total counts of each cell, then multiplied all values by the median of the total count distribution across cells. This ensured all cells had the same total counts. We then centered each feature (gene) to have zero mean and scaled to have unit standard deviation prior to running PCA or nonlinear dimension reductions such as tSNE  or UMAP For the Patel dataset, scran, census, and QUMI normalizations were applied to TPM values. The QUMI target distribution was set to Poisson-lognormal with shape 2.0. Only the 5685 genes used by the original authors were included as input to both dimension reduction algorithms. We directly visualized the cells in two dimensions using PCA, GLM-PCA, and UMAP.For the Segerstolpe dataset, after excluding non-endocrine cells, QUMI normalization  was applied to TPM values and GLM-PCA was run on all 18,301 genes that had at least one nonzero count value across all cells. The number of dimensions was set to 20, and categorical batch indicator variables were regressed off from the latent factors using a linear model. We visualized the endocrine cells using t-SNE with the 20 batch-corrected GLM-PCA factors as input . We trained a random forest classifier on the 20 GLM-PCA features using labels provided by the original authors indicating cell types. We then used the classifier to predict the labels for the 41 cells the original authors were not able to cluster. We defined an unambiguous classification as one where the predicted probability of the assigned class was >0.5. For each cell type, the original authors validated the cluster identity using a marker gene. We therefore validated our classification by comparing the QUMI count relative abundances of each marker gene for newly classified cells versus the cells that were annotated by the original authors in the same category.Additional file 1 Contains supplementary figures S1–S8, and table S1.Additional file 2 Review history.


Data from several large high-throughput drug response screens have become available to the scientific community recently. Although many efforts have been made to use this information to predict drug sensitivity, our ability to accurately predict drug response based on genetic data remains limited. In order to systematically examine how different aspects of modelling affect the resulting prediction accuracy, we built a range of models for seven drugs  using data from the largest available cell line and xenograft drug sensitivity screens. We found that the drug response metric, the choice of the molecular data type and the number of training samples have a substantial impact on prediction accuracy. We also compared the tasks of drug response prediction with tissue type prediction and found that, unlike for drug response, tissue type can be predicted with high accuracy. Furthermore, we assessed our ability to predict drug response in four xenograft cohorts  using models trained on cell line data. We could predict response in an erlotinib-treated cohort with a moderate accuracy (correlation ≈ 0.5), but were unable to correctly predict responses in cohorts treated with gemcitabine or paclitaxel. These datasets contain molecular and drug response data on hundreds of cell lines, allowing for generation of predictive models. But despite the availability of such training data, our ability to accurately predict drug response still remains quite limited6. Reasons that make drug response prediction a hard problem include noise in the data, relatively low number of samples compared to the number of features (i.e. predictor variables), incomplete omics characterization, and the static nature of molecular data. Molecular data in such studies are usually acquired only before drug treatment7. Another important problem is the consistency of pharmacogenomics associations derived from different datasets. A number of studies examined agreement between the largest datasets and found that differences in experimental protocols and differences in data analysis likely contributed to the observed inconsistency13.Drug response prediction based on genomic information is an active area of research with many practical applications including drug discovery, drug repurposing, patient selection for clinical trials, and personalized treatment recommendations (e.g. in a tumorboard setting). Several large-scale cell line drug sensitivity screens have been generated by the scientific community  regression18, support vector machines (SVM)19, random forest (RF)3, neural networks and deep learning21, logical models3, or kernelised bayesian matrix factorization (KBMF)23. For a comprehensive recent review see Ali & Aittokallio (2018)24. However, no systematic exploration of model training strategies based on data from multiple large cell line screens has been reported so far. Also cell line-based models have not yet been compared to xenograft-based models. In the current study we attempt to close these gaps with the ultimate goal of improving accuracy of drug response prediction in cell lines and xenografts.A broad spectrum of machine learning methods has been applied to the drug response prediction problem: regularized regression methods 1 (CCLE), cancer therapeutics response portal2 (CTRP), genomics of drug sensitivity in cancer3 (GDSC) and the genentech cell line screening initiative4 (gCSI), together with the only publicly available xenograft screen ‒ Novartis Institutes for BioMedical Research PDX Encyclopedia25, NIBR PDXE  on predictive performance in drug sensitivity testing. We included data from three large cell line-based datasets, CCLE, GDSC and CTRP, respectively Fig. . Only se1, CTRP2, GDSC3, gCSI4 and one xenograft screen ‒ NIBR PDXE25.For modelling we used molecular, drug response, tissue type, and division rate data from 4 large cell line sensitivity screenings ‒ CCLEMolecular data included gene expression, copy number information, and mutation information. Expression data comes from microarrays (except for gCSI dataset where it comes from RNA-seq), values are continuous. Copy number information comes from SNP arrays, it’s continuous. Mutation information is derived from whole exome sequencing and is represented by binary values per gene (1/0 for presence/absence of mutation).26. Preprocessing details for each dataset are described in Safikhani et al.9. Molecular information for the gCSI dataset was taken from the supplementary data of the publication4.Molecular information for CCLE, CTRP and GDSC datasets was obtained from the PharmacoGx package (version 1.8.3)50, AUC, and viability at 1 µM  and AUC (area under the drug response curve) information was obtained from the PharmacoGx package; in particular we used values recomputed by the package from the raw data ‒ “ic50_recomputed” and “auc_recomputed”. In order to handle outlier values in the IC50 data, we truncated the distribution of IC50 values at the 85th percentile for each drug. Viability at 1 µM values were calculated from the raw drug response data in CCLE, CTRP and GDSC datasets.Drug response data included three metrics: IC µM Fig. . IC50  values from RNA-seq. Copy number information comes from SNP arrays, it’s continuous. Mutation information has been derived from RNA-seq data and represented by binary values per gene (1/0 for presence/absence of mutation). All molecular data was taken from the supplementary data of the publicationcontrol − AUCtreated), slope of the tumor growth curve, and differential slope (difference between the slopes under the tumor growth curves from control and treated mice: slopecontrol − slopetreated), see Fig. 25.We tested 4 different drug response metrics derived from raw drug response data i.e. volume change of the tumour during the treatment course between day 0 and day 21: tumour volume (at day 21), integral response .For tissue type prediction in xenografts we used tissue labels from paper’s supplementary dataThe sample size for each modelling task is provided in the Supplementary Tables Drug response prediction tasks and division rate/slope of growth curve prediction were regression tasks. Tissue type prediction tasks were classification tasks.27.For regression tasks we used two modelling methods: support vector machine  and random forest (rf) in the first and second part of our analysis, respectively. For classification tasks we used xgBoost (xgbTree)Each modelling method has its own set of hyperparameters: svmRadial ‒ sigma and C (cost); random forest ‒ mtry; xgbTree ‒ nrounds, max_depth, eta, gamma, colsample_bytree, min_child_weight, subsample.In order to select a subset of all available features for modelling we performed feature selection using only data from the training set. We evaluated each feature individually with respect to the association between a feature vector and a vector with target variables (filter feature selection). For this evaluation we used two functions from the caret package ‒ gamScores for regression tasks and anovaScores for classification tasks.28.The function gamScores fits a generalized additive model between a single predictor and the outcome using a smoothing spline basis function. For classification, anovaScores treats the outcome as the independent variable and the predictor as the outcome. In this way, the null hypothesis is that the mean predictor values are equal across the different classes. In each function a p-value for the whole model F-test is returned and is used as the score2), and concordance index. The concordance index is the rank correlation between observed and predicted data29. RMSE and R2 were calculated using the postResample function from caret package, the concordance index was calculated using “concordance.index” function from survcomp package (version 1.28.5)30.For regression tasks we used three accuracy metrics: root of mean squared error (RMSE), coefficient of determination (RFor classification tasks we used the percentage of correctly predicted samples as the accuracy metric.We split the data into training (70%) and test (30%) sets.Using the training set we performed feature selection.Then we fitted the model with N (from 10 to 500) selected features  on the training set data. In order to select hyperparameters, 30 different combinations of them were tested on training data via cross-validation procedures, and then the combination of hyperparameters that provides the best accuracy was used for fitting the final model.We applied the model to the test set, and calculated the accuracy metrics.We repeated steps (1–4) ten times and got average values of accuracy metrics.The following procedure was used for model testing for each  combination see Fig. :We split50, AUC, Viability_1 µM; and (3) number of features used in the model: 10, 50, 200, and 500. To test these options we built a number of models for seven drugs ‒ erlotinib (EGFR), paclitaxel (β-tubulin), lapatinib (EGFR), PLX4720 (RAF), sorafenib (RAF), nutlin-3 (MDM2) and nilotinib (ABL/BCR-ABL). These seven drugs were selected because they were in overlap between CCLE, CTRP, and GDSC screens.In order to find an optimal strategy for training drug response models, we explored the model parameter space in terms of (1) molecular types of features: expression only vs. expression + copy number + mutation; (2) metrics of drug response: IC2 and concordance index are shown in Figs. Results  in terms of R2 values we repeated the analysis for Lapatininb using a linear regression model with only one predictor ‒ expression of the ERBB2 gene, which is a known biomarker for Lapatinib response in breast cancer31. The resulting average R2 values were 0.04, 0.25 and 0.27 for IC50, AUC, and viability_1 µM metrics, respectively.In order to provide a reference for R50, nutlin-3 ; AUC, nutlin-3 and paclitaxel ; viability_1uM, paclitaxel. In cases where prediction accuracy is low, predicted and observed values are hardly correlated and therefore data points are not grouped around the dotted line.We plotted predictions against observed values for each drug response metric for Nutlin-3, Paclitaxel and Sorafenib. In each plot the data points correspond to a test set from one particular (random) train/test split Figs. . A dotte2 for modelling with all genomics data (0.153) was just slightly higher than for modelling with only expression data (0.145). While these differences are small for most of the drugs, for nutlin-3 and PLX4720 they are a bit more pronounced .We found differences in accuracy for models based on different drug response metrics, IC50 metric (average = 10) and higher for AUC and viability at 1µM metrics  within each drug response metric. The number of common features is relatively small for the IC2 and concordance index values for each drug (for AUC models) are shown in Figs. 2 between 0.2 and 0.3, while nilotinib and sorafenib showed the lowest average predictability . Average R2 for each tissue separately is shown in Fig. The average R1 and with the performance from integrated (combined) random forest method (CRF)32, which was the second top performing method in the DREAM drug response prediction challenge33 . Corresponding R2 values are shown in Table We compared our results for the CCLE dataset with the performance of elastic net models from the original CCLE study4 as an in-vitro training set. It is a high-quality pharmacogenomic dataset reasonably consistent with the CCLE and GDSC datasets. We used the NIBR PDXE25 data as an in-vivo validation set since this is the only publicly available xenograft screen. We assessed 6 modelling tasks in each set, i.e. prediction of tissue type, doubling time/slope of the tumor growth curve, erlotinib response (lung samples), gemcitabine response (pancreas samples), paclitaxel response (breast samples), and paclitaxel response (lung samples). Tissue type and doubling time prediction tasks serve as a positive controls ‒ we assume that these phenotypes should be explained by genomics data better than drug response.We used the gCSI studyFor tissue prediction and doubling time/slope of the untreated tumor growth curve we used the top 400 features with the lowest p-values from F-test (see Feature selection subsection in Data and Methods section). For drug response prediction (since sample sizes were much lower) we used just the top 100 features.In the xenograft tissue prediction we had 5 tissue classes with 27–50 samples per tissue. In the cell line tissue prediction we tried modelling with 13 tissue-classes with 10–68 samples per tissue, and with the 6 largest tissue classes (each tissue had at least 23 samples).2 and concordance index values.Prediction accuracy results for all prediction tasks are collected in Table In addition to tissue prediction tests performed using all available cell lines and xenografts we made a test with equal number of samples per tissue (16 samples in training set and 7 samples in test set). A heatmap of the confusion table for the case of cell line predictions is shown in Fig. While the differences in accuracies between tissue type prediction and doubling time/slope prediction are consistent for cell lines and xenografts, drug response prediction accuracies for the same drugs are not consistent between cell lines and xenografts. Particularly, the best performing drugs are gemcitabine and paclitaxel (breast) for cell lines and erlotinib, paclitaxel (lung) for xenografts. The level of consistency is also illustrated by the number of common molecular features between cell lines and xenografts out of the top features pre-selected for modelling. There are 31 common features out of the top 400 for tissue prediction between cell lines and xenografts, only 4 common features out of the top 400 for doubling time/slope prediction, and almost no common features out of the top 100 for drug response prediction.In order to make the quality of prediction across different regression prediction tasks visually accessible, we plotted observed versus predicted values for different prediction tasks. In each task the data points correspond to a test set from one particular (random) train/test sets split Fig. .Figure 5Manual inspection of outlier cases from observed vs. predicted plots Fig.  shows thWe tested how well the models trained on cell line data can explain drug response in xenografts. For that we 1) trained drug response models using cell line genomics and drug response data (AUC), 2) got model predictions using xenograft molecular data as inputs, 3) assessed the correlation of resulting predictions (in cell line AUC units) with the actual xenograft drug response. We tried two strategies with respect to training set composition ‒ training using all cell lines and training using only cell lines that match the tissue type of the corresponding set of xenograft samples. Results in terms of correlation coefficients as well as predicted vs. observed plots for this analysis  are shown in Figs. Among four xenograft response metrics used in this study, volume and slope are expected to be positively correlated with cell line AUC while integral response (ΔAUC) and differential slope are expected to be negatively correlated. For both training strategies  we managed only for erlotinib to get predictions with the right sign of correlation coefficient (between predictions and observed drug response) and substantial absolute value for all xenograft drug response metrics. Volume and slope metrics worked especially well in the case of erlotinib, with correlation about 0.5 in both cases.We have shown that, dependent on the cellular phenotype that we want to predict from genomic data, the accuracy of prediction varies substantially. Tissue type classification can be achieved with relatively high accuracy. The percentage of correctly predicted samples is 0.79 for the cell line set, and 0.89 for the xenograft set. The accuracy of prediction depends substantially on the size of the training set for each tissue class. When we additionally include tissues which have from 10 to 20 samples into our cell line modelling set, the average accuracy drops from 0.79 to 0.64.While the accuracy of tissue type classification is high on average, it varies across tissues. In a series of tests where we use equal number of samples per tissue (16 samples in the training set and 7 samples in the test set), we found that we have the lowest accuracy for lung samples, higher accuracy for breast samples, and the highest accuracy for pancreas, colon, skin and blood samples Fig. . Interes2cell lines = 0.17, R2xenografts = 0.19. The lower accuracy of prediction shows that, unlike for tissue type, there is less information about the rate of cell division in the static expression data, which can be due to the post-translational regulation of the activity of cell cycle proteins. Additionally the prediction problem is complicated by the fact that expression data originates from the pooled group of cells that were at different cell cycle stages prior to expression profiling.The second cellular phenotype we aimed to predict was the cell line doubling time, or slope of untreated tumor growth curve in case of xenografts. Here we got lower accuracy compared to the tissue type prediction. The average accuracy is quite consistent between cell lines and xenografts: RThe main phenotype of interest in our analysis is drug response. Analogously to the doubling time prediction case, the accuracy of drug response prediction is substantially lower than accuracy of tissue type prediction. A possible reason for that is (as in the case of doubling time) that the static gene expression data obtained prior to drug treatment doesn’t necessarily contain enough information to fully explain drug response .2 for random forest models (tested on gCSI data) is comparable to average R2 for support vector machine models : R2RF = 0.23, R2SVM = 0.19. Also according to our results the number of top features selected for modelling (via filter feature selection), within the tested range of 10–5000 features, doesn’t influence the resulting accuracy of predictions, which rather depends on the strength of correlation between top feature(s) and the outcome .We see that the accuracy of drug response prediction varies across the drugs, e.g. in our tests based on CCLE, CTRP and GDSC datasets average R33. However, the use of additional omics profiles, e.g. methylation and proteomics data can still improve the accuracy of prediction34. Also it was recently shown that transcriptional perturbation signatures (e.g. from the LINCS-L1000 project35) can be successfully used for drug response prediction36.We found that the average accuracy of models based on all genomic data (expression + mutation + copy number data) is just marginally higher than the average accuracy of models based on expression data only Fig. , which s37. Additionally we compared these three metrics with a recently proposed GR_AOC metric which takes into account the difference in growth rates between cell lines38. GR_AOC performed slightly worse than AUC and viability_1uM but better than IC50  than those which additionally take into account data from untreated controls ‒ “Integral response” and “Differential slope”, with an average R2 = 0.095 and 0.145, respectively.The choice of a drug response characterization metric has a serious impact on the accuracy of predictions. Having compared three drug response metrics for cell lines we found that the area under the drug response curve (AUC) provides the highest predictive performance. AUC combines information about drug efficacy and potency into a single value, and it was reported to be the most robust metric previouslyC50 Fig. . Compari2 for models trained on cell lines from all tissues (n = 329) was 0.267, while for models that used only cell lines from a certain tissue for each drug (n = 26–68), the average R2 was 0.102. Low number of samples makes it harder to distinguish the true signal from noise33, especially in the case of multi-omics datasets which are high-dimensional. To deal with high-dimensionality in our modelling tests, we employed filter-based feature selection (i.e. selected a subset of features that have high association with a target variable). Alternative options for combating high-dimensionality include literature based feature selection (e.g. using the list of cancer census genes from COSMIC for modelling24), and common dimensionality reduction methods like PCA. Creating aggregated features using pathway information39 can help with dimensionality reduction, however in our preliminary tests we found that combining pathway features with simple genomic features (selected via feature selection) can improve resulting accuracy only marginally  with the tumor volume and the slope of the tumor growth curve. Performance of the models built and tested on xenograft and cell line data separately can’t explain why this type of prediction worked only for erlotinib-treated cohort. Indirectly it shows that pharmacogenomic associations were consistent between cell line and xenograft dataset only in the case of erlotinib, which has only one target-molecule, EGFR, but not in the cases of more pleiotropically acting drugs such as gemcitabine and paclitaxel which block DNA synthesis and cell division, respectively. Also differences in cell growth and differences in drug response quantification between 2D and 3D model systems probably contributed to the inconsistency in pharmacogenomics associations between cell lines and xenografts.While more and more data from high-throughput drug sensitivity screenings become available, accurate drug response prediction remains challenging. As we show in the present study, we cannot expect that accuracy of drug response prediction in cell lines (or xenografts) will be equal (or even close) to accuracy of tissue type prediction. Nevertheless it is possible to fit models of drug response which will provide moderate accuracy. Here we explored the parameter complexity one faces when fitting these models. We found that generally the choice of a particular machine learning algorithm doesn’t influence accuracy, and the number of variables included in the model doesn’t matter much. What matters, though, is the degree of association between the top predictor variables and outcome. Among the parameters that do influence accuracy are class of molecular predictors ‒ we see that expression data has the highest explanatory power, response metric  and, quite importantly, the number of samples in the training set.40. In the present study we tried to predict drug response in xenografts using models trained on cell line data, which can be seen as an approximation to the patient prediction scenario. We managed to get reasonably accurate predictions only in the NSCLC xenograft cohort treated with erlotinib, but not in other cohorts treated with either gemcitabine or paclitaxel. With more xenografts and patient material screens publically available in the future, it will be possible to understand in which cases drug response associations are transferable between cell lines and xenografts/patients and in which cases they are not. This understanding, combined with the wealth of available high-throughput data, will bring closer the era of personalised cancer medicine.Mastering cell line drug response models will make it ultimately possible to perform personalized prediction of drug sensitivities for individual cancer patientsSupplementary materials.