Comprehensive molecular profiling has revealed somatic variations in cancer at genomic, epigenomic, transcriptomic, and proteomic levels. The accumulating data has shown clearly that molecular phenotypes of cancer are complex and influenced by a multitude of factors. Conventional unsupervised clustering applied to a large patient population is inevitably driven by the dominant variation from major factors such as cell-of-origin or histology. Translation of these data into clinical relevance requires more effective extraction of information directly associated with patient outcome.survClust, an outcome-weighted clustering algorithm for integrative molecular stratification focusing on patient survival. survClust was performed on 18 cancer types across multiple data modalities including somatic mutation, DNA copy number, DNA methylation, and mRNA, miRNA, and protein expression from the Cancer Genome Atlas study to identify novel prognostic subtypes.Drawing from ideas in supervised text classification, we developed CDKN2A deletion across cancer types. Visualization of somatic alterations, at a genome-wide scale  and at the individual gene level, using circomap further revealed indolent versus aggressive subgroups in a pan-cancer setting.Our analysis identified the prognostic role of high tumor mutation burden with concurrently high CD8 T cell immune marker expression and the aggressive clinical behavior associated with survClust software tool is available at https://github.com/arorarshi/survClust.Our analysis has revealed prognostic molecular subtypes not previously identified by unsupervised clustering. The algorithm and tools we developed have direct utility toward patient stratification based on tumor genomics to inform clinical decision-making. The  Cancer is a complex disease with heterogeneous clinical outcomes. Understanding how patients respond to treatment and what drives disease progression and metastasis is critical for managing and curing the disease. Linking comprehensive molecular profiling data with patient outcome carries great promise in addressing such important clinical questions. This requires innovative statistical and computational methods designed for integrative analysis of multidimensional data sets to model intra-tumor and inter-patient heterogeneity at genomic, epigenetic, and transcriptomic levels. Each of these molecular dimensions is correlated yet characterizes the disease in its own unique way. In order to arrive at a comprehensive molecular portrait of the tumor, multiple groups have proposed statistical and computational algorithms to synthesize various channels of information including methods developed by us  and A complexity of applying unsupervised learning to molecular phenotypes lies in that it does not necessarily lead to unique answers. This limitation is well understood in the field of text learning. Consider the problem of clustering a collection of documents where multiple data substructures can be present including authorship, topic, and style. The outcome of the clustering is likely driven by a mixture of these underlying structures. As a result, there is often no single “right” answer in unsupervised clustering problems. In most complex data applications, many local optima exist that poses special challenges in optimization. Xing et al.  proposedDrawing analogy with the text learning problem described above, the molecular profile of a tumor is influenced by a multitude of factors including cell-of-origin , histolosurvClust algorithm as a supervised learning approach that aims to identify cancer subtypes that are not just molecularly distinct but also prognostically significant. It is an outcome-weighted integrative clustering algorithm for survival stratification based on multidimensional omics-profiling data. The algorithm learns a weighted distance matrix that downweights molecular features with no relevance to the outcome of interest. This method can be used on individual platforms alone, or by integrating various molecular platforms, to mine biological information leading to distinct survival subgroups. In this study, we analyzed over 6000 tumors across 18 cancer types. Each disease type was classified by survClust based on six molecular assays—somatic point mutations, DNA copy number, DNA methylation, mRNA expression, miRNA expression, protein expression, and the integration of the six assays. The results have revealed novel survival subtypes not previously identified by unsupervised clustering.To overcome the current limitation of molecular clustering analysis, we developed the The analysis in this study was conducted on the Cancer Genome Atlas dataset. This included six molecular data types: somatic point mutations, DNA copy number, DNA methylation, mRNA expression, miRNA expression, and protein expression across 6209 tumor samples covering 18 cancer types. Data pre-processing and normalization procedures are described in Additional file Xm be the mth  data type of dimension Nm (number of samples in the mth data type) by pm (number of features in the corresponding data type). Rows are samples and columns are molecular features. Data types may consist of continuous  or binary (mutation status) data. Overall survival is defined as the time from diagnosis to death or last follow-up. The number of samples in each tumor type and the number of molecular features in each data type are summarized in Additional file Let  mth m = ,…,M dataa and b, the weighted distance [For a pair of two samples distance  is calcua and b are feature vectors of length p for samples a and b, respectively, and W is a p × p diagonal weight matrix with W = diag {w1, …, wp}. Samples are close to each other when the value of dw is small and dissimilar when dw is large.where wj  are obtained by fitting a univariable Cox proportional hazards model fitted for each feature:The weights t represents the survival time, xj is the jth column of matrix X of length N, h0 is the baseline hazard function, β is the regression coefficient, and exp(β) is the hazard ratio (HR).where w. An HR = 1 corresponds to the null that the feature is not associated with survival. This is reflected in a log (1) = 0 weighting in the distance matrix. Since W is a diagonal matrix with diagonal elements wj , we can simply use Euclidean distance for computing distances if we transform the data as follows:We consider the absolute value of HR on the logarithmic scale as the weight Euclidean distances are sensitive to scale of the observations so after incorporating weights, we standardize the data by its grand total:X′, with i rows (N samples) and j columns (p features). Then, one can compute the pairwise distance between samples a (i = 1) and b(i = 2) as:where D is calculated for all pairwise samples across M data types. All samples having full survival information are kept, and the union of all samples (Nunion) across M data types is utilized when analyzing a wide number of samples. Non-overlapping samples in data types are added as NA to have a uniform set of Nunion samples.Conversely, a weighted distance matrix The integrated weighted distance matrix is calculated by averaging over the weighted distance matrices:Iw averages the inter- and intra-sample similarity profiles over the M data types. Iw is then processed by survClust via classical multidimensional scaling (MDS) [Iw follows the Euclidean norm and hence can be represented in n − 1 dimensions. The strong assumption of the Euclidean norm is also important for k-means, as it is essentially trying to assign samples to the closest centroid or calculating the sum of squared deviations from centroids.where ng (MDS)  and clusng (MDS) . Classicng (MDS)  providedjth gene is mutated in the ith sample, and 0 otherwise. A challenge with the mutation data matrix is the sparsity. It is known that somatic mutation data exhibit a long-tailed distribution in which a relatively small number of variants appear in tumors frequently while the vast majority of variants occur extremely infrequently. We consider genes that are mutated in > 1% of the samples. After incorporating weights, this data is no longer binary, but it still remains sparse. Due to such data sparsity, computing the Euclidean distance is not appropriate and may lead to inflated distance measures [Somatic mutation data is represented as a binary data matrix where each entry is coded as 1 if the measures . To combN (samples) by p (genes). Then, the pairwise distance between sample vectors a and b is calculated as follows:Let  see Eq.  of dimenwherew01 = sum of weights of p features that are zero in sample vector a but non-zero in sample vector b;w10 = sum of weights of p features that are non-zero in sample vector a but zero in sample vector b;w11 = sum of weights of p features that are non-zero in sample vector a and non-zero in sample vector b.dw is a proportion of the sum of effect sizes in which only one is non-zero among those in which at least one is non-zero [Note that non-zero .survClust classifies sample populations by incorporating outcome information. Cross-validation was used to prevent overfitting and arrive at more generalizable solutions. The cv.survclust function performs cross-validation for the desired number of folds and outputs cross-validated solution labels. In the data analysis, we performed 5-fold cross-validation in the following steps: (1) Split the data into 5 random partitions, label 4 of them as the training sets and the remaining one as the test set. (2) The weighted distance matrix was calculated from the training data set alone (Eq. survClust clustering was performed to arrive at outcome-weighted labels in the training set. (3) Each test set sample was assigned a class label based on its molecular feature vector and weights derived from the training set. We note the survival information for the test sample was not used in assigning its class label, ensuring an unbiased assessment of survival association for the class assignment on the test set. (4) Step 2 was repeated until predictions were made on all 5 test data sets across all 5 folds. (6) Clusters were tracked by centroid relabeling  = ∑ Vk.where k is the one with the maximum logrank statistic.For a two-group comparison, the logrank statistic follows a chi-square distribution with 1 degree of freedom. A value greater than 3.84 is considered statistically significant at an alpha of 0.05. The optimal Here we calculate the pooled within-cluster sum of squares and standardize it by the total sum of squares similar to methodology used in the gap statistic  to seleck clusters C1, C2, …. Ck, with Cr denoting the indices of observations in cluster r, and nr = |Cr|. LetSuppose that the final labels have clustered the data into wr is the sum of all pairwise distances in cluster r, {ij} represents a pair of samples belonging to a cluster Cr, and Iw is calculated from Eq. where Ws decreases monotonically as the number of clusters k increases. The optimal number of clusters is where Ws is minimized and creates an “elbow” or a point of inflection, where addition of more clusters does not improve cluster separation. Another property of Ws is that it can be used to compare among different datasets as it lies between 0 and 1 after standardization. This is useful for comparing survClust runs between individual data types and when we integrate them.Here The Rand index is used to measure agreement between two classification labels. When this Rand index is adjusted for chance, it is called the adjusted Rand index. The Rand index and adjusted Rand index have a maximum of 1 and a minimum of 0. Here 0 means the two data labels have no shared information and 1 means they are the same labels.k for a survClust run. In Fig. survClust fit for a cluster k based on training data from each fold was used to predict cluster membership for the test data. Final sample labels were aggregated over all folds and cluster meaning was preserved across folds via centroid relabeling  is of interest, a more supervised approach is needed. We demonstrate this using a simulated data example Fig. : Fig. S1a, b) denote a pair of sample vectors measured for p features and W is a diagonal weight matrix over p features with W = diag {w1, …, wp}. The weights wp ′ s are obtained by fitting a univariable Cox proportional hazards model for each feature in the training data with repeated training-test sample splits for cross-validation . This step reduces the computation considerably by transforming the problem from sample by feature to sample by sample. Note that different sample sizes across data types are allowed, i.e., a sample can be measured for some but not all platforms. Next, the weighted pairwise distance matrices are integrated by summing overweighted m data types (combineDist), which retains all samples with at least one data type available, with complete pairwise information. survClust then projects the integrated and weighted distance matrix into a lower dimensional space via multidimensional scaling (MDS) and then clusters sample points into subgroups via the K-means algorithm. More details can be found in the “Our algorithm allows the integration of multiple data modalities. Given ace Fig. d, the alsurvClust to the TCGA data set including 6209 tumor samples in 18 cancer types to identify survival outcome-associated subtypes defined by somatic mutation, DNA copy number, DNA methylation, mRNA expression, and protein expression, individually and integratively. A summary of the sample sizes and feature space is included in Additional file survClust integrated subtypes versus those derived from unsupervised clustering methods commonly used in TCGA studies including COCA and iCluster. The logrank statistic compares estimates of the hazard functions of each subgroup to the expected values under the null hypothesis . Larger logrank statistic suggests stronger evidence of survival association. For example, although unsupervised classification of the liver cancer samples elaborates on molecularly distinct subtypes, they do not exhibit significant survival separation  [survClust integrative classification, on the other hand, identified subtypes that are molecularly distinct and also show significant survival difference   exhibit the most prolonged survival times followed by IDH-mutant tumors without the codeletion (IDH-mutant-non-codel), with IDH-WT tumors demonstrating more aggressive clinical behavior. We performed survClust on 6 available molecular platforms  in 512 LGG samples as profiled by the TCGA. The optimal number of clusters k was chosen by assessing survClust fits over logrank test statistics and standardized pooled within-cluster sum of squares in cross-validation (see the “Low-grade gliomas (LGG) have a unique molecular footprint, characterized by e genome . As showrognosis . IDH-mutsee the “” sectionsurvClust solution for LGG was optimized at k = 5, with the IDH-mutant-codel (c3) and IDH-mutant-non-codel (c1) subtypes associated with good prognosis as expected  showed association with poor survival, enriched for mutations in EGFR and PTEN gene and concurrent chromosome 7 gain and 10 loss, resembling glioblastomas. Interestingly, survClust identified a small IDH-mutant subtype characterized by CDKN2A deletion (c4) that showed markedly worse survival among the IDH-mutant tumors, similar to the IDH-WT group (c5) that tends to behave far more aggressively with prognosis similar to glioblastomas. In addition, a copy number quiet subgroup (c2) was identified that showed high expression of mir-1307 and mir-29c   and the subset of head and neck cancer (HNSC) with elevated TMB. The DNA mismatch repair (MMR) signature tracks high TMB subgroups in stomach cancer (STAD), endometrial cancer (UCEC), and colon cancer (COAD). The APOBEC signature is prevalent in the bladder (BLCA) and cervical cancers (CESC). Finally, the aristolochic acid signature (signature 22) is enriched in a liver cancer subgroup identified by survClust  and a hypermutated microsatellite instability (MSI) (c4) subtype [panelmap in Fig. P < 0.001) and predominantly carried mutants in ARID1A, PIK3CA, and PTEN genes. The c1 subtype, consisting of primarily high-grade serious tumors, was associated with the worse outcome with a 5-year survival of 58% compared to 95%, 84%, and 83% for c2 (POLE), c3, and c4 (MMR), respectively, and characterized by higher frequency of mutations in TP53, PPP2R1A genes, low TMB, and older patients with serous endometrial tumors (60%). The c3 subtype was characterized by a higher frequency of CTNNB1 mutants. Immune cell decomposition data derived using the CIBERSORT [P < 0.001) , intermediate (c3), and low (c2) mutation burden. The c1 subtype was associated with good outcome, high TMB, high neo-antigen load, high APOBEC load, and high expression of the CD8 T cell immune marker (P = 0.002)   and showed concordance with MLH1 silencing status. A similar subdivision of the low TMB group by TP53 mutation status was seen where c3 carried TP53 mutant samples unlike c2. Correlation with histology revealed significant enrichment of the mucinous adenocarcinoma subtype in c1 and c2  compared to c3 . In addition to the hypermutated subtypes of endometrial, bladder, and colorectal cancers, we also observed high TMB subgroups with concurrently high expression of CD8 T cell markers in the cervical cancer c1 subtype  emerged in various cancer types , CCNE1 (c5) and AKT2(c6) amplification in ovarian cancer, and MDM2 amplification (c4) in sarcoma  was also associated with poor survival . Among cancer gene deletions, CDKN2A loss defined multiple subgroups associated with poor survival including papillary kidney cancer (c1), low-grade glioma (c4), lung adenocarcinoma (c4), and soft tissue sarcoma (c1) , which indicates influence by mRNA features toward integration  to evaluate the concordance between different solutions. RI is calculated as the proportion of sample pairs that are assigned together in the same cluster in one solution versus another, adjusted for what is expected by random chance. It provides an indirect measure of how much a particular data type contributes to the integrated solution. A non-zero adjusted RI across solutions would suggest shared biology across assay types in some tumors. For example, the mutation subtypes of endometrial cancer Fig. h have thion Fig. a. Classik > 2 solutions. The k = 4 integrated solution clearly resulted from effective integration of multiple data types including DNA methylation, DNA copy number, and mRNA expression with adjusted RIs of 0.33, 0.26, and 0.25, respectively . The k = 5 integrated solution performed the best among the 6 platforms with larger contributions from mRNA (RI = 0.63), copy number (RI = 0.62), and mutation (RI = 0.57) . miRNA, mRNA, and copy number showed high median logrank statistics over rounds of cross-validation demonstrating their role as potential prognostic classifiers.The integrated survClust analysis stratified papillary kidney cancer type into 3 groups, with CN sharing maximum information with the integrated solution (adjusted RI = 0.32), followed by mRNA (0.31), miRNA (0.24), RPPA (0.23), and methylation (0.19). Lower grade glioma displayed a wide range of variability among platform type in terms of the logrank statistic  in which sample points are organized into subgroups via the K-means algorithm. Our analysis showed that the integrated analysis outperformed the individual data types in multiple cancer types, highlighting the importance of the multi-omic approach. Finally, we developed various visualization tools including panelmap and circomap that greatly facilitate the interpretation of the results.Our approach is innovative in several aspects. First, it directly incorporates survival information in the clustering approach by utilizing weighted distance matrices. Specifically, molecular features are weighted by their corresponding log-hazard ratio (logHR) estimated from univariable Cox regression from training data. As a result, informative features will have large weights in clustering whereas noninformative features will have weights close to zero and thus minimal influence on the clustering. When unsupervised clustering fails to identify survival-associated subtypes, we show that our supervised analysis allows effective extraction of survival information and leads to clinically relevant molecular stratification. Secondly, survClust based on six molecular assays—somatic point mutation, DNA copy number, DNA methylation, mRNA expression, miRNA expression, protein expression, and integration of the aforementioned six assays.As more clinically annotated genomic data becomes available as a result of clinical sequencing programs , 37, oursurvClust analysis of copy number data alone identified aggressive clinical behavior of tumors with CDKN2A deletion in multiple cancer types, and poor survival for a subtype of ovarian cancer carrying aberration on Chromosome 19. Furthermore, the integrated clustering analysis via survClust shed light on clinical subtypes that were not identified by individual platform alone. For instance, the integrative analysis of the glioma using survClust revealed a small novel subtype characterized by CDKN2A deletion and IDH1/2 mutation  as the clinical endpoint of interest. A future extension of the algorithm will be needed for molecular stratification associated with other types of outcome such as treatment response which requires a modification of the weighting scheme appropriate for binary and categorical types of outcome. Secondly, the current integration across multiple data types does not allow different weights for the distance matrices computed from each individual data modality. A weighted matrix integration may further allow more flexible integration of the different data types. Also, the integrated results presented in the paper were run on a large feature space : Linux, Mac OS X, WindowsProgramming language: RLicense: GNU, GPLAny restrictions to use by non-academics: NoneAdditional file 1: Supplemental File containing all Supplemental Figures, Supplemental Tables, and Supplemental Note including additional details on the method and data analysis. Table S1. Supplementary Table summarizing input data of 18 cancer types across number of samples analyzed and total number of features that went into clustering for mutation, Copy Number (CN), Methylation, mRNA expression, miRNA and Protein. Table S2. Supplementary Table showing comparisons of survClust integrated solution versus unsupervised clustering results from published TCGA studies. Table S3. Summary of survival association for the survClust Copy Number solutions versus unsupervised TCGA solutions using the logrank statistic. Table S4. Summary of survival association for the survClust Methylation expression solution versus available TCGA solutions using the logrank statistic. Table S5. Summary of survival association for the survClust mRNA expression solution versus available TCGA solutions using the logrank statistic. Table S6. Summary of survival association for the survClust miRNA expression solution versus available TCGA solutions using the logrank statistic. Table S7. Summary of survival association for the survClust Protein expression solution vs available TCGA solutions using the logrank statistic. Table S8. Cross tabulation of the ovarian cancer copy number survClust class labels vs AKT gene Copy number changes. Figure S1. survClust simulation analysis. Figure S2. A second simulation example. Figure S3. survClust integrated solution of low grade glioma (LGG). Figure S4. survClust identifies tumor mutation burden (TMB) patterns across cancer types. Figure S5. CD8 expression stratified by the survClust mutation subclasses across various cancer types. Figure S6. survClus tidentifies copy number alteration patterns across cancer types, Global view of Copy Number Changes. Figure S7. survClust identifies FGA patterns across cancer types that associate with survival (Kaplan Meier curves). Figure S8. survClust integrated solution across six molecular platforms in various cancer types. Supplementary Note Description of data pre-processing, simulation details and the centroid re-labeling approach.


Molecular classification of cancers has the potential to automate and decrease errors in cancer classification. We previously showed that transcriptomic classification is comparable to methylomic and mutation methods for glioma classification and may provide benefit in predicting survival prognosis. Here we validate the transcriptomic classification method on a single molecule counting gene expression platform using formalin-fixed paraffin embedded samples.p = 0.005) with median survival time of 12.1 months, compared to non-TP1 groups. Network analysis revealed that cell cycle genes play an important role in distinguishing TP1 from non-TP1 cases and that these genes may play an important role in glioma survival. This could be a good clinical pipeline for molecular classification of gliomas.Gene expression profiling has been shown to be comparable to other molecular methods for glioma classification. We sought to validate a gene-expression based glioma classification method. Formalin-fixed paraffin embedded tissue and flash frozen tissue collected at the Augusta University (AU) Pathology Department between 2000–2019 were identified and 2 mm cores were taken. The RNA was extracted from these cores after deparaffinization and bead homogenization. One hundred sixty-eight genes were evaluated in the RNA samples on the nCounter instrument. Forty-eight gliomas were classified using a supervised learning algorithm trained by using data from The Cancer Genome Atlas. An ensemble of 1000 linear support vector models classified 30 glioma samples into TP1 with classification confidence of 0.99. Glioma patients in TP1 group have a poorer survival (HR (95% CI) = 4.5 (1.3–15.4),  IDH) [IDH genes further refines the histological classification of primary glioblastoma  [IDH mutated, IDHmut) [Gliomas, are neoplasms arising from the cerebral hemisphere , which eIDH) . These m, IDHwt) ,11 compa IDHmut) . Molecul IDHmut) ,11,12.There is a 14% disagreement rate between histology and molecular testing . The intMethylation-based classification is another method that automates tumor classification and may identify worse prognosis subgroups ,14. VerhWe recently reported a transcriptomics-based classification scheme for gliomas . The CanIDH classification. RNA profiling can be processed from multiple platforms, such as RNA sequencing, microarray, PCR, single molecule counting, and others, whereas methylation profiling mostly uses microarray platforms in the clinical setting [TM platform over PCR for its efficiency at validating multi-gene panels and over RNA sequencing for its lower cost for gene panels with lower gene numbers and over both PCR and RNA sequencing platforms for its lower hands-on time, ease of use, and readiness for clinical translation [Both methylome and TP based platforms can reduce pathologist time and processing resources for many institutions which use histological analysis. Implementation of TP may be beneficial as an alternative or supplement to methylome and  setting . We sele setting  single mnslation . MoreoveMolecular profiling methods like methylomics and transcriptomics may have limitations as well. Both methods require more tissue than traditional histological methods. Obtaining more tissue may not be feasible in some patients. Additionally, biases may exist in trained algorithms which rely too heavily on one data center or profiling platform.Pathology archives store and have ready access to formalin fixed paraffin embedded (FFPE) tissues, making this a rich resource for RNA profiling and classification. Storage of flash frozen tissues is becoming common practice due to the increasing popularity of molecular tests. A gene expression quantification platform which can readily accept both sample types would be ideal for clinical applications. The nCounter technology facilitates a rapid, sensitive, and reproducible transcriptomic profile from a complex mixture of RNA obtained from both FFPE and flash frozen tissues. This technique has been successfully applied to define candidate markers for acute myeloid leukemia , breast We describe our initial evaluation of nCounter analysis of custom genes in a brain cancer cohort at Augusta University. We present our results on the technical reproducibility of the technology. Our study demonstrates validation of 168 gene classifier  developeThis was a single-institution, retrospective study examining brain glioma tissue samples from patients seen at Augusta University (AU) Medical Center. The formalin fixed paraffin embedded (FFPE) tissue blocks were obtained from the Department of Pathology. The clinical and demographic information was obtained from the Georgia Cancer Center (GCC) Cancer Registry.We queried the Cancer Registry for all potential “brain cancers” with a pathology specimen ID. All archived tumors extracted from “Brain” at Augusta University were requested from the Cancer Registry. Only the cases with archived tissue and with a histological diagnosis of “astrocytoma, oligodendroglioma, oligoastrocytoma, or glioblastoma” were kept. We found 296 potential brain tumors archived between 2000 and 2018 under these criteria . This inWe successfully retrieved seven flash frozen glioma tumors and 117 FFPE glioma tissue blocks for RNA isolation. We used 2 mm diameter cores of the FFPE tissue to target areas with high tumor content in the tissue block. These cored locations and histologic classifications were verified by a board-certified pathologist. In total, one block was cored in three separate locations, 49 blocks were cored in two separate locations, and 67 blocks were cored in one location. The seven flash frozen tissues were cored if the tissue was greater than 4 mm in diameter, otherwise all of the tissue was used for RNA extraction.Of the isolations, 89 RNA specimens from FFPE blocks and seven RNA specimens from flash frozen tissue were run on the nCounter. After quality control (QC) of the RNA, gene quantification data from the nCounter, 34 runs were removed because of errors in positive spike in controls or normalization. On further histologic review by board-certified pathologists, 14 tissues did not meet our inclusion criteria. Forty-eight samples were used for classification, including 41 FFPE and seven flash frozen tissue samples. Thirty-nine samples were successfully classified .n = 48) are presented in n = 10), glioblastoma (n = 27), oligoastrocytoma (n = 4), and oligodendroglioma (n = 7) (n = 30), TP2A (n = 4), TP2B (n = 4), and TP3 (n = 1).The clinical and demographic characteristics of the AU cohort ( (n = 7) . We applPrior to large-scale analysis, day to day reproducibility of the pipeline was evaluated using seven FFPE samples. Reproducibility was evaluated starting from extraction of RNA to completion of the nCounter analysis. The single molecule count data were used to evaluate the within day and between days reproducibility analysis. We found high day-to-day replicability (average pairwise correlation 0.987) of our gene expression quantification platform from six RNA samples run 48 h apart .When count data were plotted against the replicates, the data lined up along the regression line with a tight distribution A. We alsWe previously reported on our ensemble algorithm which combines 1000 linear support vector classifiers trained from TCGA RNASeq data clustered using our combined UMAP and density-based clustering algorithm . We applThe ensemble model classification using 1000 linear support vector classification models  for the p = 0.005, To examine survival difference, TP1 cases were compared to non-TP1 cases. As expected, TP1 patients have worse survival than non-TP1 cases  inhibitors, and cyclin-dependent kinase inhibitors . Of the IDH mutation and 1p/19q codeletion data  and comparable to gene expression quantification techniques . This versatility provides an advantage over methylomic classification approaches which relies on a microarray-based platform. These data taken together make the Nanostring platform a strong candidate for classification and prognostication for gliomas ,44,45,46We collected 34 formalin-fixed paraffin embedded (FFPE) tissue blocks from the Augusta University Medical Center Department of Pathology from specimens archived between 2002 and 2018. We obtained 7 flash frozen brain cancer tissue from the Georgia Cancer Center Tumor Tissue and Serum Biorepository. Inclusion criteria included histologic diagnosis of oligoastrocytoma, astrocytoma, oligodendroglioma, anaplastic oligodendroglioma, or glioblastoma. Specimens were excluded if they were biopsy samples. No statistical methods were used to predetermine sample size. The investigators were blinded to allocation during experiments and outcome assessment.Demographic and clinical data were collected through the AU Cancer Registry and validated through the patients’ electronic health records. Data and sample collection were conducted through a consent-waived retrospective arm of an Institutional Review Board approved study (Biomarkers and Therapeutics in Cancer). The study was conducted according to the Declaration of Helsinki (1996) and was approved by the institutional review board at the Augusta University. Overall-survival was used as the clinical endpoint for the analysis of the data.For all FFPE tissue blocks collected, a board-certified pathologist reviewed the corresponding H&E stained tissue section, classified the tissue according to the WHO guidelines, and identified regions with >60% tumor nuclei. A 2 mm diameter core was then punched from the identified region of the tissue block and dispensed in a 96-well 2D matrix barcode system for sample storage and tracking.We developed a customized pipeline for processing of FFPE cores onto the nCounter system. Cores and flash frozen tissue were mechanically and chemically disrupted using a bead homogenizer and Citrisolve  to deparaffinize and disrupt the FFPE tissues. RNA was then extracted from the lysate using a column-based RNA extraction kit . Qualtiy of isolated RNA was assessed with a Tapestation 2000 and Nanodrop. Concentration of RNA was assessed using bioanalyzer for RNA fragments greater than 300 bp in length. The RNA concentration used for loading amount on nCounter was the lower RNA concentration between the Tapestation calculation and the Nanodrop calculation. RNA with concentration less than 20 ng/uL was not used for downstream analysis. A total of 200 ng RNA were hybridized and loaded on the nCounter.All statistical analyses were performed using the R language and environment for statistical computing .We submitted 168 genes (including 8 housekeeping genes) to Nanostring Technologies [The output files from nSolver were read into R for further QC, normalization, and data processing. We normalized the captured transcript counts using multiplicative normalization factors calculated with geometric means to first the codeset’s internal positive controls and then the geometric mean of the reference genes included in our assay . The fully normalized counts were log2-transformed.TCGA Glioma gene expression data, which contains both RNAseq and gene expression microarray data combined through Empirical Bayes, was downloaded from Ceccarelli et al. 2016. The final dataset contained 1032 samples and 12,717 genes. TCGA Glioma data were centered and scaled. Clinical data were downloaded from the same source and matched to the processed TCGA Glioma data.Data from Nanostring and TCGA RNAseq were integrated using the Empirical Bayes-based Combat algorithm  implemenOne hundred and sixty-eight genes were identified for use in supervised classification using two complementary methods. First, significantly differentially expressed genes amongst our four groups were identified through linear models for microarray data (LIMMA) analysis  and 26 gIn order to decrease the potential of overfitting, data were split into four folds, where three folds were trained on the unsupervised model classes and the supervised models predicted on the remaining fold. In this way, no sample was used for both training and making classification calls. This results in calls for each sample from 1000 linear SVC models. Models were combined into one ensemble model using a plurality voting method which reports the most popular class and the proportion of the 1000 LSVC models which agree on this most popular class. A confidence score is calculated by taking the proportion of models classifying samples into the most popular class divided by the proportion of models classifying samples into the second most popular class. If the confidence score is greater than 3, then the ensemble model classifies the sample into the most popular class. If the confidence score is less than or equal to 3, than the ensemble model prediction is “ambiguous”.p < 0.05, all p values were two sided. Patients with no history of recurrence or death were censored at the date of last follow-up visit. Patients who died of natural causes unrelated to cancer were censored at time of death.We modeled over-all survival with Kaplan–Meier and Cox proportional hazards and tested for significance with the log rank test all using the “survival” R package. Kaplan-Meier survival plots were made through the “survminer” package. The statistical significance of differences was set at We performed LIMMA  to identWe describe a pipeline for RNA isolation and classification of glioma tumor tissue from formalin fixed and flash frozen tissue. We describe a machine learning algorithm that can classify glioma gene expression data from at least three different platforms  into a clinically and prognostically relevant subtype. Thus, regardless of the RNA profiling platform, our method can predict glioma subtype. Network analysis indicates cell cycle and cell proliferation pathways represent a key difference between TP1 groups compared to the others.


Biological networks are representative of the diverse molecular interactions that occur within cells. Some of the commonly studied biological networks are modeled through protein-protein interactions, gene regulatory, and metabolic pathways. Among these, metabolic networks are probably the most studied, as they directly influence all physiological processes. Exploration of biochemical pathways using multigraph representation is important in understanding complex regulatory mechanisms. Feature extraction and clustering of these networks enable grouping of samples obtained from different biological specimens. Clustering techniques separate networks depending on their mutual similarity.We present a clustering analysis on tissue-specific metabolic networks for single samples from three primary tumor sites: breast, lung, and kidney cancer. The metabolic networks were obtained by integrating genome scale metabolic models with gene expression data. We performed network simplification to reduce the computational time needed for the computation of network distances. We empirically proved that networks clustering can characterize groups of patients in multiple conditions.We provide a computational methodology to explore and characterize the metabolic landscape of tumors, thus providing a general methodology to integrate analytic metabolic models with gene expression data. This method represents a first attempt in clustering large scale metabolic networks. Moreover, this approach gives the possibility to get valuable information on what are the effects of different conditions on the overall metabolism. Biological data produced by high throughput experiments, and in particular by Next Generation Sequencing technologies are being accumulated in publicly available databases. Multi-year research projects, such as The Cancer Genome Atlas (TCGA) , are proIn the case of Genome Scale Metabolic (GSM) models, this integrative approach has been used in several studies –5. It usad-hoc distance based on Shannon-Jensen divergence is used to compute their pairwise distances. Since the number of nodes in such networks is in the order of thousands, we explored a simplification technique [Following these ideas, we decided to devise a clustering technique for network data. Again, each patient is represented in form of a network integrating gene expression data and a GSM model. We obtain a representation of the networks based on a probability distribution of their shortest paths, and an echnique , 8 that echnique  can be uThe overall work-flow used in the present study is schematically shown in Fig. https://portal.gdc.cancer.gov) were downloaded in the form of FPKM normalized read counts. As summarized in Table Breast Microarray dataset contains 418 samples of four intrinsic molecular subtypes based on BreastPRS [Breast RNAseq dataset contains 401 samples of two intrinsic molecular subtypes based on PAM50 [Lung dataset contains 337 samples divided into three groups: Adenocarcinoma (159 samples), Squamous carcinoma (150 samples), and Solid tissue normal (28 samples). The Kidney dataset contains 299 samples divided in three groups: 159 samples of clear cell Renal Cell Carcinoma (ccRCC or KIRC), 90 samples of Papillary Renal Cell Carcinoma (PRCC or KIRP), and 50 samples of Solid tissue normal.Gene expression data of breast cancer from microarray experiments are publicly available in the NCBI Gene Expression Omnibus database   were generated considering the metabolites as nodes, with edges connecting metabolites involved in the same reaction, one as a reagent and the other one as the product, as illustrated in Fig. The same metabolite can have more than one cellular compartment localization fulfilling different functions; thus, we considered it as a different metabolite in each compartment Reactions not catalyzed by any enzyme were not considered and disconnected nodes were excluded as well. Indeed, networks representing different patients differ only for the weights of their links, and this information would be absent in both the above cases.The recurrent metabolites T, where, for all i, qi,j=1 if node i is in cluster j and qi,j=0 otherwise). The summarized network k clusters into a supernode; the weight matrix Ws is obtained by Ws=QTWQ. Further details are given by the authors in [Given a network ribed in . Brieflythors in .k clusters, we adopted spectral clustering; a well know algorithm used for cluster analysis [In our approach, network summarization is applied to the largest connected component of each network  package in R . For eactransition matricesi to be reached in one step by a random walker located in node j. Therefore, The final step of our approach involves the calculation of distances between the summarized networks, in order to cluster the patient samples. In , we reprk supernodes of the summarized networks, we defined the network distance Given two networks dJS is a metric known as Jensen-Shannon distance [where distance , definedibutions . Using tmentclass2pt{minimGram matrix or distance matrix.The square matrix containing in each row the vector representing a sample from the dataset is usually called the For obtaining the final clustering results of the proposed approach, we applied spectral clustering from the sklearn package in Python  to the dk of clusters that were set  or maximized (↑) to have more accurate results (column Better if). Matlab scripts used for clustering evaluation are provided as Additional file Several metrics exist for clustering. In the experiments, we consider an extended set of metrics often adopted for clustering evaluation , 35–38, Performance results for the four datasets are reported in Table For the Breast Microarray dataset, we observe that clustering gene expression data leads to poor overall performance. Indeed, RI values below 50% reveal very low accuracy and negative ARI values reveal that accuracy is even lower than the one that could be obtained by a random partitioning. At the same time, more than 50% of the times the wrong decision is taken (MR=52.72%). Similarly, all the other metrics confirm poor performance. This can be ascribed to the microarray technology used to quantify the gene expression, which is known to be less sensible for slight differences in expression measurements compared to the RNA-seq method used for the other datasets. Moreover, Bartlet et al.  investigThis also holds true for the case of the Breast RNAseq dataset, even though with a small improvement as compared to the case of gene expression data.For the Lung and Kidney datasets, clustering of raw data leads to quite high performance, as witnessed by low values of MR and high values for all the other performance metrics. Comparable performance is achieved using whole graphs, slightly better than using summarized graphs.For all the datasets, the execution times for computing the distance matrices for summarized graphs strongly decrease (≈ 35 times less of the execution times for whole graphs).Therefore, we can conclude that the proposed simplification proves to be beneficial for clustering “difficult” high-throughput data  and only minimally detrimental in the other cases. In all cases, it allows a strong reduction in execution times, making it feasible for big data analyses.T-distributed Stochastic Neighbor Embedding (t-SNE) is a nonlinear dimensionality reduction technique that allows embedding of high-dimensional data for visualization in a low-dimensional space . It modeFigure vice versa. Moreover, a small of group of embedded points from the two carcinoma classes appears close to the solid tissue normal class. These few visual anomalies also reflect in the clustering results (right column), where the three clusters appear more uniform that the ground truth , consistently with the visual judgement.The top row of Fig. The middle row provides the visual representation for the whole graph data. Here, the two carcinoma clusters are spatially contiguous, without marked separation (left column). The same can be said for the clustering results (right column), where most of the misclassified points appear spatially close to the cluster they are deemed to belong to . This shows that spectral clustering on these data fails like we would fail in judging based on the t-SNE visual representation.The bottom row reports the visual representation for the summarized graphs. Similarly to the case of gene expression data, three spatially separated clusters can be identified, corresponding to the three different classes (left column). Only few carcinoma samples lye in an ambiguous region of the plane, at the intersection of the three clusters. Clustering assigns almost all of these few samples to the ccRCC class (right column), predicting the ambiguous plane region as belonging to this class.The above analysis of visual representations obtained using t-SNE suggests that the proposed simplification leads to clusters that are better separated than in the whole graph case, and similarly to the gene expression case, thus confirming the comparable performance achieved. Analogous analyses carried out for all the considered datasets  set of Matlab scripts, used for clustering evaluation.Additional file 7 t-SNE-based visual representations. The file AdditionalFile7.pdf provides the t-SNE visual representations for expression, whole graph, and summarized graph data in Breast Microarray, Breast RNAseq, and Lung datasets.Additional file 8 Heatmap representations. The file AdditionalFile8.pdf provides the heatmap representations for summarized graphs in Breast Microarray, Breast RNAseq, and Lung datasets.


Unfortunately, current studies often focus on the optimization of isolated model characteristics instead of examining the overall modeling workflow and the interplay of the individual model components. Moreover, they are often limited to specific data sets only. Therefore, they are often confined by the irreproducibility of the results and the non-transferability of the approaches into other contexts. In this study, we present a thorough investigation of translational models and their ability to predict the drug responses of cancer patients originating from diverse data sets using the R-package FORESEE. By systematically scanning the modeling space for optimal combinations of different model settings, we can determine models of extremely high predictivity and work out a few modeling guidelines that promote simplicity. Yet, we identify noise within the data, sample size effects, and drug unspecificity as factors that deteriorate the models’ robustness. Moreover, we show that cell line models of high accuracy do not necessarily excel in predicting drug response processes in patients. We therefore hope to motivate future research to consider in vivo aspects more carefully to ultimately generate deeper insights into applicable precision medicine.Translational models directly relating drug response specific processes that can be observed  In the context of personalized medicine, finding genomic patterns in a cancer patient that can predict how a specific drug will affect the patient’s survival is of great interest. Translational approaches that directly relate drug response specific processes observed in cell line experiments to their role in cancer patients have the potential to increase the clinical relevance of models. Unfortunately, existing approaches are often irreproducible in other applications. In order to address this irreproducibility aspect, our work comprises a thorough investigation of a diverse set of translational models. In contrast to other approaches that focus on one isolated model characteristic at a time, we examine the overall workflow and the interplay of all model components. Additionally, we validate our models in multiple patient data sets and identify differences between cell line and patient models. While we can establish models of high predictive performance, we also expose the deceptive potential of optimizing methods to a specific use case only by showing that those models do not necessarily depict biological processes. Thus, this study serves as a guide to interpret new approaches in a broader context to avoid the dissemination of noise-driven models that fail to serve in everyday applications. The avaThus, cell lines were not only extensively characterized by means of their molecular profile, such as mutational status, gene expression, proteomics, copy number variation or methylation, but also based on their responses to broad panels of drugs –7. ConseWhile these efforts are great in comparing and improving distinct components of a model, such as batch effect correction methods , featurein vitro data. Even though studies have shown that cell lines reflect many important aspects of human in vivo biology, they also exhibit significant differences. This is especially evident regarding the absence of an immune system and a micro-environment of tumor cells in cell cultures, making the direct translation of preclinical models into a clinically relevant context rather difficult [in vitro and in vivo mechanisms already in the model building process [in vitro and in vivo data via batch-effect removal concepts [Furthermore, in the majority of these studies, the ultimate goal is to understand the molecular mechanisms in living patients, while the analyses are performed on ifficult . Transla process , can helconcepts . Howeverin vivo processes and therefore hope to motivate research to consider in vivo aspects more carefully to ultimately generate deeper insights into applicable precision medicine.Accordingly, in this study, we intend to systematically investigate the workflow of translational models of such kind, using our R-package FORESEE  to explo50 values, a remove unwanted variation (RUV) homogenization between the cell line- and patient data set, a landmark gene filter, no further feature preprocessing of the remaining gene expression features, and an elastic net or a lasso regression algorithm, which both accomplished a performance of AUC of ROC = 0.99  values, with RUV4 as homogenization method between training and test features, including all genes as features without further feature preprocessing and a random forest algorithm for model training yields the best performance of all 3,920 modeling pipelines in both the GSE51373 data set (AUC of ROC = 0.859) and the Erlotinib subset of GSE33072 (AUC of ROC = 0.897). Despite this outstanding performance in two of the data sets, the pipeline fails to exceed an AUC of ROC of 0.56 in all other data sets, confirming the more general notion that a perfect model for a specific data set is insignificant for other use cases.From the comparison of the settings of the best model in each of the cell line—patient data set scenarios in From our systematic scan of model space, it becomes clear that the problem of reproducibility does not only occur for the very best pipeline in each specific data set, as shown in the previous section. This problem manifests for almost all modeling pipelines, independent of the absolute value of their performances.Section A in the upper left corner of the correlation plot in In order to put the poor reproducibility into perspective and examine the problem in a broader context, First, even though all of the models were trained on cell line data from GDSC, the training data sets differed enormously. This is owed to the fact that the drugs included in the GDSC database are not screened on all of the available cell lines, but mostly on smaller subsets. Consequently, the training data sets for the different translational scenarios in this paper varied in sample size and in the distribution of the cell lines’ tissues of origin depending on the drug that was modeled .none, variance and pvalue produced feature sets that could differ in size by a factor of 3, which makes the comparison of these methods in two different data sets unbalanced.Second, the features that were available for training the translational model varied significantly. Both, the microarrays that were used in the generation of the patient gene expression data and the gene IDs that were originally chosen to define the features, have an impact on the resulting overlapping feature list for each distinctive pair of cell line- and patient data set. After converting the reported gene names into Entrez Gene IDs  with bionone, which simply skips any type of homogenization between the training and test set, can result in very different conditions for different patient data sets. Consequently, it is no surprise that some of the pipelines do not show a high correlation among the different patient data sets.Third, while the training sets’ distributions of gene expression values were highly similar in all seven settings —which isTaking into account these effects of the data constitution, it becomes accessible that the correlation of the ensemble of 3,920 pipelines is not in a perfect range. Still, they do not justify the large extent of the bad correlation that was attained in this study and the lack of transferability of these approaches in general.Concluding from the weak correlation of model performances among different data sets and the widespread performance distributions of the set of 3,920 pipelines in one individual data set, the translational models that are investigated in this study seem to be affected by noise. In order to assess the extent of this noise, we investigated three different aspects that could potentially compromise modeling performance.As apparent from the violin plots in One principal difference between the patient data sets is their sample size. Therefore, From the direct comparison between the actual performance distributions and the random ones, it becomes apparent that certain model findings may be less significant than previously thought. In the cases of GSE9782 GPL96 and GPL97 for instance, even though the best models have comparably low absolute values of AUC of ROC, their performances are still higher than predictable by 10,000 random guesses. Thus, they effectively allow the training of robust models that are not entirely driven by noise. The Sorafenib cohort of the GSE33072 data set and the GSE18864 data on the other hand yield worse results on average when a model is trained to predict the response than when the response is guessed at random, even though the absolute values of the AUC of ROC seem more promising than the ones of GSE9782 GPL96 and GPL97. This is particularly prominent for the Sorafenib cohort of the GSE33072 data set, which exhibits a median AUC of ROC of 0.46—the lowest among all data sets.This indicates that these data sets are predominantly governed by noise and should therefore be considered with caution for the training of universally valid clinical models. For GSE18864, this finding is concurrent with previous findings by Geeleher et al. , the autEven though, compared to GSE18864 and the Sorafenib subset of GSE33072, the 3,920 FORESEE models yield a better performance distribution in the other data sets, it becomes explicit that the fitted models do not outperform the randomized models by far. Especially the large range of the performance distribution of translational models applied to patient data with permuted gene labels (medium blue) demonstrates that an optimized model of very high predictive performance in one application scenario—as listed in this study or as published separately by other groups—does not necessarily produce a model that truly captures the mechanisms that are relevant for the drug response and can therefore not be employed in clinical practice. Instead, the high predictivity can equiprobably stem from random mechanisms that were captured by the model by chance and are only relevant to that specific scenario. In this sense, our analysis demonstrates the importance of considering the possible degree of randomness that could affect modeling results, especially when the sample size of the test population is small.In an attempt to further analyze the extent of randomness accompanying the performance results of the shown FORESEE modeling pipelines, a drug specificity analysis was conducted. For each of the patient data sets and each of the 266 drugs contained in the GDSC database, a set of 100 randomly chosen pipelines, which are listed in To a certain degree, it is expectable that drugs that differ in their specific mode of action still kill the same cancer cells and therefore have a comparable effect on cancer patients, because drug action is also impacted by general characteristics, such as unspecific cell death and survival mechanisms, drug efflux or resistance properties. Still, especially in the context of searching for distinct drug targets and related fields, such as drug repurposing, the specific mechanisms of drug action are considered a major contribution towards the cell’s or the patient’s survival in the end. Yet, especially the examples of Erlotinib and Sorafenib in this analysis demonstrate that the effect of the drug that is used during the training process on model performance is merely minor. Indeed, the fact that so many models reveal good performances, even though the drugs that the models were trained with are not the ones that were predicted in the end, suggests that the internal mechanisms that dominate the translational models for drug action in cancer patients are not at all specific to the drug of interest, but rather to general mechanisms of drug response. This hypothesis is further supported by the observation that the drugs that are more predictive than the drugs of interest do not share the same mode of action. For example, Docetaxel is a chemotherapeutic drug that binds to microtubules and prevents their disassembly, eventually causing the initiation of apoptosis, while the drug occupying the first rank in predicting GSE6434 patients, Tanespimycin, is an antineoplastic antibiotic that inhibits the heat shock protein 90 (Hsp90), which then promotes the proteasomal degradation of oncogenic signaling. Likewise, Erlotinib is an inhibitor of EGFR signaling, while the drug that is ranked first for GSE33072 patients is Tozasertib, which targets aurora kinases.in vitro models that were trained and tested on cell line data in a five-fold cross-validation for all six drugs of interest: Docetaxel, Cisplatin, Paclitaxel, Erlotinib, Sorafenib and Bortezomib. In order to keep the cell line models comparable to the translational patient models, the features used for training the in vitro models were restricted to those that were available for the respective patient models. The violin plots in in vitro study point towards the notion that information about drug response and the cell death or cell survival mechanisms can in fact be captured in the underlying gene expression data and can be used to create a prediction model for all investigated drugs in this study. Consequently, the lack of transferability between the different translational models reveals that there are other, more complex challenges to overcome for the translation from in vitro processes into an in vivo setting.Following the observation that randomized models can yield prediction performances close to performances gained from actually trained translational models and the finding that translational models are not necessarily specific to the administered drug, the question arises whether the 3,920 modeling pipelines tested in this study are at all capable of predicting drug response or if all observations should be attributed to noise. In a baseline experiment, we therefore investigated As shown in the previous analyses, there is no modeling pipeline that yields outstanding prediction performance for all data sets that were investigated. Yet, it is possible to identify settings that help to generate more promising models.A closer investigation of which pipeline settings were enriched among the best model performances for each individual data set provided important insights into promising modeling choices. The top panel of Binarization of the cell response, using either a cutoff value or the k-means algorithm, is beneficial in three of the seven data sets.Reducing the features to genes of the landmark gene list of the LINCS consortium during feature selection is significantly enriched in four of the seven data sets.Reducing the dimensionality of the input data by applying principal component analysis to the gene expression values is advantageous in four of the seven data sets.There are several well-performing black box algorithms, but linear regression is significantly enriched in three of the seven data sets.While RUV homogenization yields outstandingly high-performance values for the GSE6434 data set, it is strongly underperforming in every other data set.Taking into account all categories together, it becomes explicit that settings that aim to simplify the model, such as binarizing the output data or reducing the dimensionality of the input data, tend to be significantly enriched in well-performing models and are therefore good principles to aim for.in vitro models and translational models. Strikingly, model settings that work significantly better for in vitro models, such as support vector machine regression as modeling algorithm, no transformation or power-transformation of the cell line response values (ln(IC50)), z-score transformation of the gene expression values or a gene filter based on variance, do not show superiority for translational models that aim to predict in vivo drug performance. This finding is confirmed by another independent study, in which we compared the model performances of in vitro models and translational models. In order to investigate whether models that are proven to perform well for a certain set of cell lines also perform well on another set of cell lines, as well as on patients, for each drug of interest we split the GDSC data into three parts—a training set, a validation set and a test set—and compared the performance of 3,920 FORESEE modeling pipelines on the validation set, the test set and the patient data. The boxplots in in vitro data also performs significantly better on the test bin of the in vitro data, proving that the information that is captured in those in vitro models can be applied to other in vitro use cases. If these 300 FORESEE pipelines of high performance are used to predict the response of patients in a translational model setting however, they do not show a significant improvement in 5 of the 7 scenarios. Thus, models that are selected based on their performance on cell line data alone, do not necessarily excel in predicting the drug response of patients. Likewise, model settings that yield higher prediction performance in translational models, such as a gene filter based on the landmark gene list or principal component analysis as feature preprocessing method, do not seem to be advantageous for in vitro models. In the case of linear regression, which is significantly enriched in the modeling of 3 out of 7 patient data sets, the performance in cell line data is even lower than every other modeling algorithm tested in vitro. Consequently, also the correlation between in vitro and translational models is rather low, as depicted in in vitro level are observable in a more isolated context, which allows to capture more details in more complex models, while these specific characteristics depreciate during the translation into the greater context of an in vivo setting, where drug response and survival processes are reflected in much simpler patterns.The overview provided by Despite the lack of transferability of both extremely high performing model pipelines and the general set of 3,920 pipelines with a poor correlation among different data sets, the enrichment analysis revealed that certain settings can be appointed that promote model performance. In order to validate those findings, we compared a subset of 35 modeling pipelines that followed the design rules established in this study—linear regression models fitted on genes that are part of the landmark gene list and preprocessed by applying PCA—to the overall set of 3,920 FORESEE pipelines. The ensemble of investigations of this study proves that a systematic analysis of translational models, not focussing on one specific model element only, but instead considering the whole modeling pipeline and the interplay of preprocessing methods and modeling algorithms, can be beneficial to refine the understanding of modeling coherences and key model characteristics. Tools, such as our R-package FORESEE, have the potential to support this systematic analysis by spanning an extensive space of model parameters that is searched for an optimal setting. By considering all combinations of model settings, our scan indeed spawned superior models for individual application scenarios. This superiority however was proven to not be robust across multiple data sets.While this seems to resemble the common overfitting problem for models that are trained and tested on one data set only and fail to replicate in a new, unknown one, this finding is different in the sense that during the translational model fitting process, a model is trained on cell line data and it is not until after the training process that the model is applied to the patient data to obtain a prediction. Considered in isolation, this separation of training- and test set should provide for a definite quantification of model performance that is robust with respect to overfitting to the underlying data. Strikingly however, the incorporation of the search process on the full model parameter space seems to introduce an effect that is quite similar to the overfitting problem. In this regard, we could demonstrate that random predictions could in fact yield performance values that could compete with those of actual models. Moreover, we showed that the drug that a translational model is trained with does not have a prevailing impact on the model performance. Effectively, the training drug does not even have to share the same mode of action with the drug that is predicted to create a good model.in vitro experiments in the preclinical drug development stage to learn drug mechanisms and develop predictive models that are ultimately applied to patients is the current state-of-the-art in pharmacological research. With our analysis, we would therefore like to draw attention to the pitfalls that accompany this concept. While a computational model might capture perfectly the processes that are initiated during the drug response in cancer cell lines, this model might not be suitable to predict the therapeutic outcome of patients in the clinic. In order to find reliable in vivo predictors, regularities of other cases of drug applications in patients should be taken into account to establish an estimate of confidence for the approach.Taken as a whole, this proves that translational models in the fashion of the ones developed in this study are not yet able to explain all processes of drug action in patients. On the contrary, the processes that are dominating translational models are not yet understood, such that random or drug unspecific models seem to be similarly good predictors in most of the cases. Moreover, our study revealed that modeling pipelines that are well fitted to predict cell line drug response differ considerably from translational modeling pipelines that excel in predicting patient drug response. Yet, the concept of using One concept that transpires in the investigation of translational models in this study is the superiority of models that are kept simple by for example binarization of the training output data, feature reduction to landmark genes, input dimensionality reduction by PCA or application of simple regression methods. This phenomenon coincides with the KISS principle , a well know principle in engineering and design, which promotes the avoidance of unnecessary redundancy and complexity in order to solve a problem. At the same time, our investigations demonstrate that the drug-specific processes in patients are not fully captured in simple translational models of the kind developed with FORESEE. Thus, besides keeping models simple and focussing on the acquisition of general processes of drug action and survival in the data, another promising approach could be to integrate profound and mechanistic knowledge about drug targets and patient-specific characteristics into the models.Along the way to more insightful models, also other important factors should be considered; some of which also transpired within this study. One challenge in the development process of translational models is the extent of utilizable data. Despite the fact that the generation of omics data has experienced a tremendous boost over the last years, the availability of patient data sets that include both molecular data, such as gene expression, and a measure of drug efficacy is limited. Existing patient data sets comprise small sample sizes only, which makes model training with them susceptible to noise, as apparent from the sample size effect analysis. Moreover, these data sets are highly heterogeneous in terms of the used array technologies, the measured drug response type, the chosen gene names and the applied normalization method. The creation of robust models that are easily transferable to other data sets and into other contexts requires large data sets of the same structure and with the same preprocessing, such that they are effortlessly comparable. At the same time, models that are trained on data sets of small sample sizes only, should be thoroughly tested with regard to their randomness and robustness in more than one context before being purported as reliably predictive. To that effect, this study accentuates the importance of systematic and unbiased benchmarking of new approaches.Another important aspect of the exploration of translational models is the reliability of the translational principle as such. With the great amount of noise and randomness in the results of current studies, more focus should be laid onto the investigation to what extent cell lines are actually predictive of patients and whether or not other model systems, such as organoids  or patiomics data type and at least one measure of drug response. The inclusion of RNASeq data and other data types, such as mutational status, copy number variation, methylation and protein expression could significantly improve the predictive performance of models, simply by adding more levels of information. Since not all biomedical mechanisms are fully shown on the gene expression level, other data types could provide valuable insights to generate a detailed picture of the disease and drug mechanisms.Lastly, missing predictivity of the translational models shown here could be partially ascribed to the molecular data type that the models were trained on. So far, the investigations of translational models were mainly focused on gene expression data of microarrays owed to the absence of patient data sets that comprise more than one R [Analyses of this paper were conducted with the software R .50  values as drug response measure were derived from GDSC data [ForeseeCell object. The data base contains pretreatment gene expression data measured with Affymetrix Human Genome U219 arrays of 1,065 different human cancer cell lines stemming from diverse tissues.For the training of the translational models in this study, both gene expression data and logarithmized ICDSC data  and formForeseePatient objects including gene expression data and one measure of in vivo drug efficacy, which is summarized in For the testing of the translational models in this study, information of patients with breast cancer  or clinical partial response (cPR), were manually classified as responders, whereas patients, whose clinical response had been categorized as stable disease (SD) or progressive disease (PD), were classified as non-responders. Similarly, the responses to Bortezomib of the GSE9782 data set were manually classified into responders, if the response had been reported as complete response (CR), partial response (PR) or minimal response (MR), and non-responders, if no change (NC) or progressive disease (PD) had been reported.Previous studies have shown an increased predictivity of translational drug response models whose training sets do not only include cell line samples from the same tissue of origin as the tumor, but also cell lines originating from diverse other tissues . TherefoIn order to provide unique features and avoid any mismatches, all gene names that occurred more than once were removed from both the training and the test object.For the homogenization of the gene expression data and the batch effect correction between cell line train- and patient test data, seven different methods were chosen:ComBat from the package sva [The method kage sva  used emplimma [The method limma  removed none offered to employ modeling pipelines without any correction.In order to have a thorough investigation of the usefulness of homogenization methods, the option quantile [The method quantile  homogeniRUV4 [The method RUV4  homogeniRUV4 , which aRUV applied a self-implemented function that was inspired by the function RUV4 [princomp of the stats package of R applied a principal component analysis on the gene expression of the housekeeping genes [lm of the stats package of R on the first 10 principal components. The residual gene expression of all genes was considered as homogenized data.The method ion RUV4  and a fuion RUV4  in orderng genes  of both YuGene [The method YuGene  applied Selecting a smaller subset out of the thousands of genes in microarray data to increase the robustness of biological models is a widely studied topic. In this paper, we compare four simple approaches:landmarkgenes reduced the features to the list of landmark genes that were determined as being informative to characterize the whole transcriptome [The method criptome .variance used the function var of the stats package of R to calculate the variance of each gene across all samples of the training data set. The 20% least variant genes were removed from the analysis.The method pvalue calculated a student’s t-test with the function t.test of the stats package of R between the most sensitive and the most resistant samples of the training data set. The 20% genes with the highest p-values were removed from the analysis.The method all considered all overlapping genes between the train- and test object without further feature selection.The option For the standardization and transformation of the features, four different methods were chosen:zscore_samplewise transformed the raw gene expression (GEX) value of each gene, by subtracting the mean GEX of all genes of one sample and subsequently dividing the resulting value by the standard deviation of the GEX of all genes of that sample.The method zscore_genewise transformed the raw gene expression (GEX) value of each gene, by subtracting the mean GEX of this gene in all samples and subsequently dividing the resulting value by the standard deviation of the GEX of this gene in all samples.The method PCA used the function prcomp of the stats package of R to convert the possibly correlated raw gene expression features into a set of linearly uncorrelated principal component features. The first 10 principal components were chosen as a dimensionality-reduced feature set for the model.The method none included the raw gene expression features without any preprocessing for the purpose of baseline comparison.The option All model pipelines in this study were based on gene expression data only, since the patient data sets that were chosen for this study did not contain other molecular features.For the final step in the model development, seven different algorithms were chosen to train a model that predicts the drug response of cell lines based on baseline gene expression profiles:linear used the function lm of the stats package of R to fit a linear model to the training data.The method lasso [α = 1) and a regularization parameter λ determined by a 10-fold cross-validation.The method lasso  fitted aelasticnet [α = 0.5 and a regularization parameter λ determined by a 10-fold cross-validationThe method asticnet , 43 fittrf [The method rf  used a rrf  to fit trf_ranger [The method f_ranger  establisridge [The method ridge  fitted aridge .svm [The method svm  trained pROC package [For the investigation of the performance of the different modeling pipelines, the area under the receiver operating curve between the true and the predicted patient classes was calculated using the  package .t.test of the stats package of R applied a two-sided student’s t-test to the two distributions.In order to investigate whether two distributions were significantly different from each other the function phyper of the stats package of R was applied to the data.In order to investigate the enrichment of certain model settings in the overall distribution of model performances with a hypergeometric function, the method roc function of the pROC package [ggroc function of the ggplot2 package [Receiver operating curves were calculated with the  package  and plot package .pearson measured the Pearson correlation with the function cor of the stats package of R.In order to determine the linear correlation of the performance distributions, the method FORESEE R package [biomaRt package [Within the context of collecting the different data sets and enforcing them into the same ForeseeObject formats for the  package , the bio package , 23 was heatmap.2 function of the R package gplots [Heatmaps were plotted with the e gplots .hist function of the graphics package of R.All histograms were plotted with the ggplot + geom_violin function of the ggplot2 package [Violin plots were created using the  package .corrplot.mixed function of the corrplot package [The correlation plot was created using the  package .S1 FigPortrayal of translational models that used the FORESEE package to train models on GDSC cell line data and subsequently predicted the drug response of patients from GSE18864, GSE51373, GSE33072 Erlotinib cohort, GSE33072 Sorafenib cohort, GSE9782 GLP96 cohort and GSE9782 GLP97 cohort. The settings for the respective best modeling pipelines can be found in (TIF)Click here for additional data file.S2 Fig(A) Distributions of the gene expression values of the cell line training data sets. (B) Distributions of the gene expression values of the patient data sets implemented as ForeseeTest objects. (C) The cell line training data sets’ composition of tissues of origin. Distributions are shown for seven different patient data sets: GSE6434, GSE18864, GSE51373, GSE33072 Erlotinib cohort, GSE33072 Sorafenib cohort, GSE9782 GLP96 cohort and GSE9782 GLP97 cohort.(TIF)Click here for additional data file.S3 FigDistributions of the performances of 3,920 cell2cell models trained on GDSC cell line data for six different drugs: Docetaxel, Cisplatin, Erlotinib, Paclitaxel, Sorafenib and Bortezomib. Each violin plot shows the performance distribution of 3,920 modeling pipelines trained with cell response information of the respective drug. The data represents the mean performance resulting from a 5-fold cross-validation.(TIF)Click here for additional data file.S4 Figin vitro validation set (red), the in vitro test set (yellow) and the patient data set (medium blue), versus the performance distribution of a subset of FORESEE pipelines on that data set, which were determined by choosing the 300 best pipelines from the in vitro validation set for six different drugs: Docetaxel, Cisplatin, Erlotinib, Paclitaxel, Sorafenib and Bortezomib.Boxplots showing the performance distribution of all 3,920 FORESEE pipelines on the (TIF)Click here for additional data file.S1 Table100 random FORESEE pipelines were chosen and then trained on all 266 available drugs in the GDSC database to predict the patient data sets.(XLSX)Click here for additional data file.S1 FileThe folder includes all relevant FORESEE model settings and modeling results presented in this manuscript.(ZIP)Click here for additional data file.


Machine learning has been utilized to predict cancer drug response from multi-omics data generated from sensitivities of cancer cell lines to different therapeutic compounds. Here, we build machine learning models using gene expression data from patients’ primary tumor tissues to predict whether a patient will respond positively or negatively to two chemotherapeutics: 5-Fluorouracil and Gemcitabine.We focused on 5-Fluorouracil and Gemcitabine because based on our exclusion criteria, they provide the largest numbers of patients within TCGA. Normalized gene expression data were clustered and used as the input features for the study. We used matching clinical trial data to ascertain the response of these patients via multiple classification methods. Multiple clustering and classification methods were compared for prediction accuracy of drug response. Clara and random forest were found to be the best clustering and classification methods, respectively. The results show our models predict with up to 86% accuracy; despite the study’s limitation of sample size. We also found the genes most informative for predicting drug response were enriched in well-known cancer signaling pathways and highlighted their potential significance in chemotherapy prognosis.Primary tumor gene expression is a good predictor of cancer drug response. Investment in larger datasets containing both patient gene expression and drug response is needed to support future work of machine learning models. Ultimately, such predictive models may aid oncologists with making critical treatment decisions. The goal of personalized medicine is to tailor treatments for individuals based on unique characteristics of their genetic background. Given the vast variety of cancers and the inherent molecular heterogeneity of the disease, personalized medicine in cancer can be particularly effective . By studWhile personalized oncology has shown signs of promise, not all cancers have such well-defined targetable pathways . This haHere, we attempt to develop an improved model for predicting drug response in patients. We apply several well-established machine learning techniques to address the technical challenges. First, to reduce dimensionality we utilize optCluster , an R paThe utility of drug response models built from in vitro data is often limited, because of the genetic and environmental differences between cell lines and patient-derived samples. Not until recently, has in vivo data been publicly available for research. For the first time, an extensive compendium of pan-cancer datasets has been released by The Cancer Genome Atlas (TCGA), which provide both clinical drug response data and gene expression profiles from primary tumor patient samples spanning multiple cancer types . In thisOur study is based on data obtained from the National Cancer Institute’s TCGA database . TCGA prWe report the results of the most accurate clustering algorithm from optCluster in Table In Fig. The ROC curves in Fig. P = 2.4 × 10–5) and B cell activation (P = 9.9 × 10–5). For a list of the top 20 pathways based on the percentage of genes within that pathway refer to the S2 Table, Additional file Once we established an optimal model for predicting drug response, we performed a gene set enrichment analysis using PANTHER . We analIn our study we have shown that primary tumor gene expression can be a good predictor of cancer drug response. By utilizing different clustering and classification methods we predicted cancer drug response with validation accuracy of up to 86%.Our analyses show the GCB model performs stronger than the 5-FU model Table . We attrWe infer from our results that some drugs target mechanisms that are shared across most cancers, while others may target mechanisms specific to certain cancer families. Our GCB pan-cancer model predicts all cancer types at comparable levels to that of the overall accuracy. On the other hand, a much higher variation in accuracy is seen from the 5-FU model . Chemokines direct trafficking and migration of immune cells and inhibition of these proteins has been proven effective in preventing the accumulation of leukocytes near sites of inflammation [While biologically informed models have been shown to be successful –30, we dammation . We beliThe results of our final approach for prediction cancer drug responses were conclusively accurate and, more importantly, interpretable. Our classifier selected genes that are integral parts of drug metabolism and cancer biology. This combination of accuracy and interpretability has been difficult to achieve in predictive models attempted in the past. We attribute our success to the utilization of in-vivo gene expression data, which eliminates the need to extrapolate human drug responses from cell line or other in-vitro features. Furthermore, our implementation of optCluster and random forest provided us with a method to perform dimension reduction in a biologically informative manner. Feature ranking, as we have shown, selects biologically relevant genes, that may yield new therapeutic targets. While recent discussion has suggested machine learning can appear as “alchemy” , we encoWe focused our study on pan-cancer models. Pan-cancer models were shown, by a previous study , to out-To determine accuracy, we used the mean value for each gene cluster, hereunto gene module, as a feature for prediction. The number of observations (58 for 5-FU and 92 for GCB) was much lower than the chosen number of clusters (~ 200), so feature selection was performed. We used the machine learning technique, random forest, to determine feature importance. Random forest utilizes a forest of decision trees to split the data into multiple subsets based on predictive power. To stabilize the ranked list of feature importance, we took the mean Gini value for each gene module after 200 random forest runs. The mean decrease of Gini index was selected because it is the recommended performance metric for imbalanced data .The algorithm was run with N number of clusters .Random forest was used to determine feature importance and perform feature selection.For each top n most important features (where n is between 1 to N), random forest was used to classify each patient.The highest classification method was logged for the best value of N and n for each clustering method and for each drug model.The best model from the previous step was selected and the parameters were tuned to capture any additional accuracy.To find the best clustering algorithm for our study, we performed clustering and classification in a five-step process  on the accuracy of the model. All approaches yielded worse results.The final 5-FU and GCB models were trained on their respective pan-cancer data in which half of the patients from the most populous cancer, STAD or PAAD respectively, were held out as validation sets Fig. . To reduOur hypothesis stated the selected genes in our final model would be enriched for pathways involved in drug metabolism or cell signaling. To test this hypothesis, we performed two statistical tests. First, we performed a PANTHER overrepresentation test. Using the results of the PANTHER analysis, we performed a t-test comparing the mean gene expression of responders and non-responders for the top twenty pathways based on the percent of genes hit against the total number of genes in the pathway (See Table S2, Additional file Additional file 1:Figure S1. Dimension reduction of genes. Figure S2. Survival data as a predictor of drug response. Table S1. Genes selected by random forest variable importance. Table S2. Top 20 PANTHER pathways in models by gene percent. Table S3. Accuracy by cancer type. Supplementary Methods Data pre-processing, gene standardization and gene selection