Glioblastoma is a rapidly fatal brain cancer that exhibits extensive intra- and intertumoral heterogeneity. Improving survival will require the development of personalized treatment strategies that can stratify tumors into subtypes that differ in therapeutic vulnerability and outcomes. Glioblastoma stratification has been hampered by intratumoral heterogeneity, limiting our ability to compare tumors in a consistent manner. Here, we develop methods that mitigate the impact of intratumoral heterogeneity on transcriptomic-based patient stratification.We accessed open-source transcriptional profiles of histological structures from 34 human glioblastomas from the Ivy Glioblastoma Atlas Project. Principal component and correlation network analyses were performed to assess sample inter-relationships. Gene set enrichment analysis was used to identify enriched biological processes and classify glioblastoma subtype. For survival models, Cox proportional hazards regression was utilized. Transcriptional profiles from 156 human glioblastomas were accessed from The Cancer Genome Atlas to externally validate the survival model.We showed that intratumoral histologic architecture influences tumor classification when assessing established subtyping and prognostic gene signatures, and that indiscriminate sampling can produce misleading results. We identified the cellular tumor as a glioblastoma structure that can be targeted for transcriptional analysis to more accurately stratify patients by subtype and prognosis. Based on expression from cellular tumor, we created an improved risk stratification gene signature.Our results highlight that biomarker performance for diagnostics, prognostics, and prediction of therapeutic response can be improved by analyzing transcriptional profiles in pure cellular tumor, which is a critical step toward developing personalized treatment for glioblastoma. Variations in histology confound results of established gene signatures.Analyzing the cellular tumor improves glioblastoma subtyping into biologically distinct cohorts.Patient risk stratification is dramatically improved by analyzing the cellular tumor.This paper describes a strategy to improve stratification of patients with glioblastoma by analyzing gene expression patterns in specific histologically defined tumor regions, and draws attention to the critical importance that intratumor histologic variability plays in the development and future clinical use of predictive and prognostic molecular biomarkers.1 Improving survival has been hindered by the inability to stratify glioblastoma by clinical parameters such as therapeutic sensitivity  and rate of disease progression. Identifying patient cohorts with similar glioblastoma tumors would improve preclinical therapeutic development, design of clinical trials, clinical decision-making, and ultimately patient outcomes.Glioblastoma is the most common and aggressive malignant primary brain tumor, with a median survival of 18.1 months.3 Current clinical stratification methods include extent of resection, Karnofsky Performance Score (KPS), age, O6-methylguanine-DNA methyltransferase (MGMT) promoter methylation, and isocitrate dehydrogenase 1 (IDH1) mutation, none of which captures the heterogeneous landscape of glioblastoma.4–9 Modern “omic” technologies, such as high-throughput genomic, transcriptomic, and proteomic profiling, enable new approaches for tumor subset identification. Omic analyses of glioblastoma samples from The Cancer Genome Atlas (TCGA) defined 4 molecular subtypes: classical, neural, proneural, and mesenchymal.10 However, these subtypes and subsequent prognostic gene signatures have not yet found clinical utility.Stratification of glioblastoma is challenging because these tumors display complex multilayered inter- and intratumoral heterogeneity.11 This methodology captures nonuniform, varying amounts of histologically diverse tissue architecture composed of cancer cells, stromal cells, vasculature, immune infiltration, and necrosis. Samples that contain a mixture of these elements may obscure detection of key tumorigenic processes enriched or depleted in specific tumor microenvironments. This histologic heterogeneity likely interferes with interpatient comparisons when biopsies are composed of inconsistent tissue architecture.We explore the hypothesis that glioblastoma intratumoral heterogeneity has impeded the development of robust molecular tools for patient stratification due to sampling regions that differ in histological structure composition. Nearly all omic studies investigating glioblastoma have used samples collected with little regard for histological structure besides necrosis.Herein we utilize an open-source glioblastoma gene expression atlas to characterize transcriptional patterns contributing to intratumoral heterogeneity and demonstrate that (1) histologic structures within a tumor are molecularly distinct and confound results of established gene signatures created from mixed-structure samples and (2) analyzing the dense cellular tumor histological structure improves glioblastoma subtyping into biologically distinct cohorts and patient risk stratification. These advances will guide the development of precision medicine approaches for glioblastoma and enhance prognostics to identify patients with the highest risk of rapid progression.n = 34 subjects) .12 Gene expression data (HTSeq-FPKM), methylation data, and corresponding clinical information from the Genomic Data Commons Data Portal for the Glioblastoma Multiforme projects  (n = 156 newly diagnosed cases).—RNA-sequencing data from newly diagnosed tumors from the Ivy Glioblastoma Atlas Project (Allen Brain Institute) normalized as Fragments Per Kilobase of transcript per Million (FPKM) mapped reads and corresponding clinical data from Swedish IvyGAP Database were used in the present analyses .13—Principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE) were performed on the 1000 most variable genes in the IvyGAP data set on log2-transformed and r = 0.92, for visualization.—A sample-to-sample correlation graph was plotted in BioLayout Express3D. Nodes represent individual samples, and edge length depicts the degree of correlation between samples with Pearson correlation coefficients above k = 1–10 was determined on the top 1000 most variable genes (factoextra R package).13 The optimal number of clusters maximizes the gap statistic.—The gap statistic comparing the total intracluster variation for K-means clustering was performed on the 1000 most variable genes in the IvyGAP data set using k = 4, the optimal clusters determined by the gap statistic analysis, and visualized by principal components (stats and factoextra R package).13—k = 4 groups (stats and factoextra R package]).13—A distance matrix on the 1000 most variable genes in the IvyGAP data set was computed using Euclidean distance measure and hierarchical cluster analysis by Ward’s method. Dendrogram construction used 14The IvyGAP data set was balanced between structures and evenly split into train and test sets using Stratified K-Folds cross-validator (n_splits = 5). Using train sets, a lasso regularized logistic regression classifier was built to classify glioblastoma multiforme (GBM) structure in independent data sets . The classifier’s cross-validation average accuracy on test sets was 98.45%. The classifier was used to assign the structure classification of all GBM-TCGA samples .15 Defaults were used for GSEA analysis, including Signal2Noise ranking metrics. Gene sets smaller than 15 genes or greater than 500 genes were excluded, and enrichment P values were estimated by 1000 permutations and corrected for multiple testing using the Benjamini–Hochberg method. Analyzed gene sets were from the molecular signature database (MsigDB), Gene Ontology (C5), Hallmark (H), or Positional (C1) collections (www.broadinstitute.org/gsea/msigdb/collections.jsp). For molecular subtypes, single sample GSEA was performed.—Gene set enrichment analysis (GSEA) was performed using GSEA software with FPKM gene expression data.www.cytoscape.org).16 For visualization purposes, clusters of functionally related enriched GO terms were manually circled and labeled, and significance thresholds were set to be highly conservative for the LE/IT and HBV/MVP structures , conservative for PNZ/PAN , and loose for CT .GSEA results were visualized using the Enrichment Map plugin for Cytoscape  were calculated following methods previously described.17 Kaplan–Meier survival analysis was performed using metagene scores to separate all IvyGAP samples into high- (metagene score >0) and low-risk (metagene score <0) groups.—Metagene scores for each sample in the IvyGAP data set .—Univariate Cox proportional hazards regression was performed using age, gender, MGMT methylation status, IDH1 mutation status, and age was performed. Each gene was assigned a hazard ratio (HR), Wald statistic, and a corresponding P value using Cox regression analysis. Genes were selected as candidates significantly associated with survival if the P value was < 0.05 (HR ≠ 1). The HR for a given gene >1 was defined as a risk gene, <1 was defined as a protective gene .—Multivariate Cox proportional hazards regression using P values were computed for the base model, iteratively fit with each candidate gene. An updated model was created adding the candidate gene with the highest log-rank test score to the base model. Iteratively, one of the remaining candidate genes was added that led to the greatest improvement. This process was continued until the concordance = 1 or 10 genes had been added .—Gene candidates from multivariate analysis were applied to the process of forward stepwise selection. Ten random seeds were generated; for each, IvyGAP CT samples were split into train and test sets using 5-fold cross-validation. Using train sets, HRs, log-rank test scores, and P value was computed. Models were excluded that had concordance <0.5 or log-rank test P value >0.05 upon prediction on the test set. The model with concordance nearest the mean (0.75) of all remaining models was selected for subsequent analyses (stats R package).—The model for each train set that underwent stepwise selection was used to predict the HR of the corresponding test set and concordance and log-rank test P value >0.05. The resulting finalized model was trained on IvyGAP CT data and used to predict the HRs for each sample in the IvyGAP CT and the entire IvyGAP data sets .—The revised model at the end of the interval validation steps was applied to the entire IvyGAP CT data set and features were excluded that had a Wald statistic R package).—The finalized survival model was applied to the entire GBM-TCGA and CT-classified data set, as above .—Either an HR of 1  or tertiles of HR values were used to classify into 2 (high-risk: HR > quantile (⅔); low-risk: HR < quantile (⅔)) or 3 groups (high-risk: HR > quantile (⅔); medium-risk: quantile (⅓) < HR < quantile (⅔); low-risk: HR < quantile (⅓)). The Kaplan–Meier method was applied to generate survival curves; significance was evaluated using 2-tailed, log-rank tests with z-score-normalized data (pheatmap R package).—When clustering was performed, transcripts and samples were organized by unsupervised hierarchical clustering using Ward’s method with the Euclidean distance metric. Heatmap visualization and hierarchical clustering were performed on log2-transformed, 18 This database is comprised of 2 companion data sets: (1) RNAseq and In Situ Hybridization data from histologically identified human glioblastoma tumor structures and (2) corresponding clinical information including patient demographics, pathology, and survival. Briefly, predefined histologic structures  were outlined on H&E stained tumor sections obtained at tumor resection, microdissected on adjacent sections, subject to RNA sequencing, and archived as FPKM mapped reads by the Allen Brain Institute , tumor invasion (LE/IT), vasculature (HBV/MVP), and necrosis (PAN/PNZ)   and J.21–23 The LE/IT structure, where the ratio of tumor cells to central nervous system (CNS) cells is low, had enrichment of normal CNS processes such as neuron development, synaptic signaling, and regulation of ion and neurotransmitter homeostasis. Thus, transcriptomic analysis of the bulk tumor edge captures CNS processes, rather than cancer specific biology. This importantly implies that if one inadvertently samples IT or LE tissue for gene expression studies, results will contain a higher contribution of expression data from normal CNS cells rather than the tumor cells, providing a misleading representation of the tumor biology. The vascular architecture (HBV/MVP) was associated with angiogenesis, regulation of blood pressure, vascular permeability, cell junction assembly, and extracellular structure organization. This region also was enriched in immune processes including regulation of phagocytosis, leukocyte migration and activation, and cytokine production, suggesting an inflammatory microenvironment. The PNZ/PAN architecture also was associated with enhanced immune processes, including monocyte and lymphocyte differentiation, and leukocyte migration and chemotaxis. Additionally, the PNZ/PAN region was characterized by biological networks associated with necrosis, cellular starvation, hypoxia, and oxidative stress.We analyzed whole transcriptome measurements of each of the 4 transcriptionally distinct structures applying GSEA of gene ontology (GO) gene sets to identify biological processes enriched in each structure relative to the rest of the tumor .15,16,21The CT structure has the highest density of neoplastic cells and the transcription profiles of this structure varied between patients. The variation between patients decreased our ability to identify biological processes associated with the overall CT structures. However, there was a trend toward enhancement of traditional cancer processes including DNA replication and repair, chromatin remodeling, and stem cell proliferation. The transcriptional variation observed in this structure, which histologically is a more homogenous distribution of neoplastic cells in all patients, suggests that analysis of gene expression profiles from CT might enable more precise identification of biologically distinct tumors. Thus, we next evaluated the expression patterns of select subtyping and prognostic gene sets in all structures and within the CT.10 Thus, this gene classifier has not translated clinically. Additionally, several analyses have reported classification of a single tumor into multiple subtypes.24–26 We reasoned that these issues might be related to histological heterogeneity within and between tumors, and may be improved by evaluating purely CT tissue.Glioblastoma molecular subtypes  are not strongly associated with clinical endpoints.10 Neural and proneural subtype-defining genes were strongly expressed in LE and IT samples, while mesenchymal subtype genes were highly expressed in HBV and MVP samples. This suggests that a biopsy taken from the tumor edge might be classified as neural or proneural, while a biopsy from the same tumor taken from a highly vascular region might be classified as mesenchymal. Subtyping all samples from each structure, demonstrated that a single patient could be classified as every subtype depending on the structure analyzed. To avoid this problem, we focused on using solely the CT, as this structure showed the most subtype gene expression variability. All 4 subtypes could be distinguished in CT  inverted the survival curve, alarmingly predicting a poor prognosis in subjects who had longer survival.We applied an established multigene predictor of glioblastoma outcome to transcriptomic profiles from structurally distinct samples and observed that predicted survival outcome was confounded by structure .17 LE ananalyzed .17 UsingMGMT status, and IDH1 mutation in the model. The final risk score calculation included MGMT status, age, and expression of 6 genes: PGAM4, ETNK2, MIA, GMPS, BCL7B, and IBSP.We next asked whether patients could be better stratified according to outcome using gene expression profiles measured for the cancer cell rich CT structure. Stepwise multivariate Cox proportional hazards regression on IvyGAP CT samples was used to create a novel glioblastoma prognostic model and risk score equation . We inclMGMT methylation status alone. Stratification of patients into moderate versus highest risk groups was statistically significant and better than MGMT status alone  creating predictive signatures for tumor sensitivity and response to treatments and (2) identifying methods to collect CT without microdissection. Using CT for clinical purposes will likely be hindered by the labor-intensive microdissection. Using macrodissection or image-guided biopsy may overcome this problem. Diffusion weighted MRI and amino acid positron tomography are already being integrated into the operating room and can localize glioblastoma regions with elevated tumor cellularity and mitotic indices prior to resection, allowing for selective CT localization and biopsy.We have shown that analysis of transcriptomics in CT can stratify patients into distinct cohorts, and that using mixed-structure samples can give misleading information. Ultimately, we believe the present study is a critical step in generating a novel set of transcriptomic-based clinical tools utilized to plan and execute optimal care for glioblastoma patients.vdaa093_suppl_Supplementary_Figure_S1Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S2Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S3Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S4Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S5Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S6Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S7Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S8Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S9Click here for additional data file.vdaa093_suppl_Supplementary_Figure_S10Click here for additional data file.vdaa093_suppl_Supplementary_MaterialClick here for additional data file.


Transcriptomics data are relevant to address a number of challenges in Toxicogenomics (TGx). After careful planning of exposure conditions and data preprocessing, the TGx data can be used in predictive toxicology, where more advanced modelling techniques are applied. The large volume of molecular profiles produced by omics-based technologies allows the development and application of artificial intelligence (AI) methods in TGx. Indeed, the publicly available omics datasets are constantly increasing together with a plethora of different methods that are made available to facilitate their analysis, interpretation and the generation of accurate and stable predictive models. In this review, we present the state-of-the-art of data modelling applied to transcriptomics data in TGx. We show how the benchmark dose (BMD) analysis can be applied to TGx data. We review read across and adverse outcome pathways (AOP) modelling methodologies. We discuss how network-based approaches can be successfully employed to clarify the mechanism of action (MOA) or specific biomarkers of exposure. We also describe the main AI methodologies applied to TGx data to create predictive classification and regression models and we address current challenges. Finally, we present a short description of deep learning (DL) and data integration methodologies applied in these contexts. Modelling of TGx data represents a valuable tool for more accurate chemical safety assessment. This review is the third part of a three-article series on Transcriptomics in Toxicogenomics. Clarifying the toxic potential of diverse substances is an important challenge faced by scientists and regulatory authorities alike . The rapIn the last years, many transcriptomics datasets have been generated to characterize the molecular MOA of chemicals, small molecules and nanomaterials exposure by transcriptomics profiling of the exposed biological systems . At the Notably, TGx approaches have been used to analyze quantitative transcriptomic data, to determine the BMD and estimate the critical point of departure for human health risk assessment ,17,18,19TGx shifts the focus from traditional end-point-driven analysis to a systems biology approach, allowing to better understand and predict the alterations in the molecular mechanisms leading to toxicity. In this context, different methods for the study of gene co-expression networks is of great interest to identify common patterns of expression among relevant genes ,4,5,6,22Furthermore, different ML methodologies have been developed and applied to the analysis of TGx datasets for the purpose of identifying toxicogenomic predictors. ML includes both unsupervised and supervised methods. The unsupervised methods, such as clustering, do not require any prior classification of the samples, grouping them based on similarities of selected features. On the other hand, supervised methods require discrete or continuous endpoints. They are often combined with strategies to identify an optimal subset of features that can discriminate the endpoint values. This subset of features can then be used for the prediction of the class or the effect of a new sample. A wide range of algorithms has been proposed to build robust and accurate predictive models, including linear and logistic models, support vector machines (SVM), random forests (RF), classification and regression trees (CART), partial least squares discriminant analysis (PLSDA), linear discriminant analysis (LDA), artificial neural networks (ANNs), matrix factorization (MF) and k-nearest neighbours (K-NN) ,24,25,26In this review, we present the state-of-the-art methodologies developed to interpret, analyze and model already preprocessed omics data. These include BMD analysis, gene co-expression network and ML methods for predictive modelling. We discuss the main ML methodologies, highlight scenarios where each methodology is most suited, the pros and cons of the different approaches, and which are the best validation strategies. We also provide a brief overview of data integration methodologies for multi-omics data analyses.One of the main goals of toxicity assessment is the study of exposure–response relationships that describe the strength of the response of an organism as a function of exposure to a stimulus, such as chemical exposure, after a certain time. These relationships can be described as dose-response curves where the doses are represented on the x-axis and the response is represented on the y-axis. From these curves, a BMD value is calculated as the dose (or concentration) that produces a given amount of change in the response rate  of an adverse effect. Normally, the BMR value is 5% or 10% change in the response rate of an adverse effect relative to the response of the control group. Furthermore, estimations of the lower and upper confidence interval for the BMD value are also computed and are called BMDL and BMDU, respectively ,30,31.In the last years, dose-response studies have been integrated with microarray technologies, thus introducing gene expression as an additional important outcome related to the dose. Indeed, the genes whose expression changes over the dose are of particular interest, since they provide insights into efficacy, toxicity and many other phenotypes. A specific challenge is to identify genes with expression level changing according to dose level in a non-random manner, identifiable as potential biomarkers .p-value.The combination of microarray technology with BMD methods results in a bioinformatic tool that provides a comprehensive survey of transcriptional changes together with dose estimates at which different cellular processes are altered, based on a defined increase in response . A classA predefined response level of interest, called BMR, is identified and the optimal model is used to predict the corresponding dose (BMD) . Moreovehttps://www.rivm.nl/en/proast) .BMD approach is also implemented by the PROAST software (/proast) , developA similar pipeline for toxicogenomics applications is implemented in the java-based US National Toxicology Program’s BMDExpress 2 tool, where a dose-response model is fitted for every gene, whose expression value is the response variable for the different doses ,40,41 T. FurtherMore recently, a novel user-friendly software based on R/Shiny, BMDx, has been introduced . In addiGene co-expression network analysis is a systems biology method used to describe the correlation patterns among genes across different experimental samples. It allows representing, investigating and understanding the complex molecular interactions within the exposed system ,45. The To understand the nature of cellular processes, it is necessary to study the behaviour of genes by means of a holistic assessment . Thus, tStudying the topology of the gene co-expression network allows identifying communities of genes that show similar behaviour. Moreover, the use of centrality measures facilitates the identification of genes that are hubs in the network . A classThe first challenge in this type of analysis is the identification of the best algorithm used to infer the gene co-expression network. Indeed, starting from the preprocessed TGx data, different methods to construct the gene co-expression network can be applied ,53,54. TVarious algorithms have been proposed based on information theory. Some of the most important ones are RelNet , ARACNE The CLR algorithm is an extension of the relevance network, but there is a correction step to eliminate false correlation and indirect effects . SimilarUnfortunately, applying different methods to the same omic dataset may not always result in consistent co-expression networks. For this reason, Marwah and collaborators recently proposed a tool, called INfORM (Inference of NetwOrk Response Module), able to infer a more stable and robust network by applying an ensemble strategy . INfORM A different approach is to infer directed graph networks, which allow not only to reveal the systematic coordinated behaviour of sets of genes but also the identification of causal and regulatory relationships between them . These mThe main assumption of read-across studies is that structurally similar compounds are likely to share a similar toxicological profile. These approaches are used to fill toxicological data gaps by relating to similar chemicals for which test data are available . TraditiTGx datasets, such as DrugMatrix , ConnectFor example, Zhu et al. developed a read-across method based on a consensus similarity approach starting from different biological data, to assess acute toxicity in the form of estrogenic endocrine disruption . MoreoveFurthermore, the toxFlow web-based application is available to perform read-across and toxicity prediction by integrating omics and physicochemical data . The impAdverse outcome pathway (AOP) is a conceptual framework that couples existing knowledge on the links between a molecular initiating event (MIE), such as contact of nanomaterial with Toll-like receptors on the cell surface, with the activation of a chain of causally relevant biological processes or key events (KE), e.g., the production of inflammatory cytokines, with the resulting adverse outcomes (AO) at the level of the organ or the organism  ,77. CoupML methods have significantly advanced in recent years and are proven to be important alternatives to experimental testing for chemicals and nanomaterials ,81,82. TML algorithms use data-driven approaches to develop predictive models. Data derived from empirical experiments is first analysed to assess its quality, and, if necessary, it is preprocessed to improve the stability of the ML models. Common preprocessing techniques include filtering out features that are not informative, removing anomalous observations  or filling data gaps. After this step, the dimensionality of the training data can still be excessive, so the data can be further preprocessed using dimensionality reduction techniques.The preprocessed data is used to train ML models that can predict a variable of interest (supervised learning) or detect patterns in the dataset (unsupervised learning). In addition to estimating parameters fitted to the data, most models also provide a set of hyper-parameters that must be optimized to achieve best performances, like the number of clusters in k-means, or the number of trees in a random forest. After training, the capability of the model to generalize beyond the training data is evaluated on an independent and identically distributed test set. In the case of multiple competing models, the best model of each family of predictors is evaluated and the optimal model is deployed in the real-world environment. A graphical representation of the process is shown in Since TGx data usually present a large number of measured molecules compared to the number of samples, they can suffer from the curse of dimensionality. Thus, the model overfitting, the spurious correlations and a trade-off between accuracy and computational complexity have to be taken into account when modelling these data .Dimensionality reduction techniques and feature selection methods can mitigate these issues and can be used in combination with ML approaches to build predictive modeling . Some exProbabilistic component modelling can be a powerful technique as it combines dimensionality reduction with highly interpretable ML models ,92. It cIf the goal of the analysis is to reduce the dimensionality by preserving the original features, feature selection approaches can be a better alternative. Indeed, it allows to reveal significant underlying information and to identify a set of biomarkers for a particular phenotype . ExampleFeature selection methods for the identification of biomarkers of toxicity can be used in combination with ML approaches to predict the toxicity of different drugs and chemicals ,101,102.Su et al. proposed a multi-dose computational model to predict drug-induced hepatotoxicity based on gene expression for toxicogenomics data . Their mTGx data are the result of an experimental measure that is prone to both technical and biological noise, due to the complexity of the exposed system. Thus, stability and reproducibility play a key role in the analysis . For exaAccording to the OECD principle of validation , one of Recently, a new methodology for feature selection from complex data, called MaNGA has been proposed . MaNGA uClustering is an unsupervised learning exploratory technique, that allows identifying structure in the data without prior knowledge on their distribution. The main idea is to classify the objects based on a similarity measure, where similar objects are assigned to the same class ,115,116.Different clustering algorithms are available and some well known algorithms are listed in Furthermore, due to the intrinsic complexity of the omics technologies and processes involved, experiments can also produce low quality data. As in other domains, the presence of outliers may strongly influence the results. Various methods exist to detect outliers and to handle them. When an outlier is detected it may simply be removed, but also other approaches are possible like substituting it with an object more similar to the others, or assigning a reduced weight ,124.E. coli subjected to toxicants at various dosages. Self-organizing map, a kind of ANN producing dimensionality reduction to a discrete space, was used as the underlying algorithm for the consensus clustering. The resulting clusters were mostly consistent with prior toxicological knowledge. Nystrom-Persson et al. [Applications of clustering in TGx include evidencing groups of samples/experimental conditions by similarity in gene expression or, analogously, classes of genes by similarity in their expression between samples/experiments. If the focus is on the responses to different stimuli, fold change similarities between pairs of readings occurred before and after the stimuli may be used as a similarity measure. Clustering may be applied to data before training a predictive model in order to tune the stratification of the samples or to train different models on different subpopulations . McNichon et al.  applied n et al.  applied In supervised learning, classification is the task of predicting the class to which a sample belongs, given the class of previously seen samples. To build such a predictive system, it is necessary to provide as training samples to a learning algorithm both the TGx features and the toxicity label. In classification tasks related to TGx data, it is common to assign to the positive class the “interesting” effects, which, for example, may correspond to events of toxicity. Different classification algorithms have been applied to the problem of predicting toxicity in the liver and kidney using TGx features, in vitro endpoints and molecular descriptors ,129,130.For example, Minowa et al. , proposeWhen training a model, the whole dataset is split into training data and test data. The training data is used to fit the model, and the test data is used to evaluate the quality of the fitted model against data unseen during training. Once fitted on the train data, the model’s generalization capabilities are estimated by multiple metrics, such as the accuracy and F1-score that are computed starting from a confusion matrix .The objective is for the model to adapt to the training data enough to be able to generalize to new data samples, but not too much to overfit and being unable to generalize to new samples. Overfitting is particularly worsened in cases where the number of samples is smaller than the number of features so that the joint distribution cannot be properly represented by the data . Many meThe more a dataset is unbalanced, the less informative some evaluation metrics become. This is because, during training, every classifier learns with less effort the negative class, at the expense of a rising rate of false negatives. Consider, as an extreme example, a dataset made of 99% negative samples and 1% positive samples. A trivial classifier that always outputs the negative label, would have an accuracy of 99% but is essentially useless.Different methodologies can be applied to compensate for imbalances in the dataset. During training, the dataset can be resampled, i.e., to reduce the imbalance in the training data, the majority class can be down-sampled (discard a given proportion of the samples), the minority class can be over-sampled, or both. Over-sampling can be as simple as randomly adding duplicate samples, or it can be a generative scheme that creates new synthetic samples combining the actual samples like hybrid methods, such as ROSE ,135, andAnother approach to reduce the effects of imbalance in the dataset is to weight differently classification errors of the classes, e.g., a false negative is ten times worse than a false positive. A trivial weighting scheme is to assign to each class the inverse of the corresponding class proportion as a weight. In conjunction to either approach or a combination of the two, after training, model evaluation should be performed using a metric that takes into account the proportions of each possible outcome such as the Matthews Correlation Coefficient ,139.Regression is a supervised learning methodology that estimates the relationship between the features (a.k.a. independent variables) and a continuous variable referred to as outcome or dependent variable. It is used in TGx to predict important quantities as the level of toxicity of a compound, the half maximal inhibitory concentration (IC50), the survival, or differences between in vitro and in vivo response.The simplest regression algorithm is the linear regression, where the outcome is predicted by a linear function of the features. Also, nonlinear regression methods are available, where the outcome depends on the features by a more complex function. E.g. Schüttler et al.  used theANNs are well known techniques for nonlinear regression . While sSince using a high number of features produces less compact models and possibly overfitting, a variety of techniques allows to select or reduce the features. Previous external knowledge can be used to select features , or theyAfter a regression model has been trained, it is necessary to internally and externally validate it. Internal validation is performed on the same data that was used in training. It measures how well the model fits the original data, but does not measure overfitting. External validation is performed on a different dataset than the one used for training. Most of the quality measures can be used for each kind of validation just changing the data the model is applied to. Prediction quality measures are extremely variegated, as potentially any similarity measure between the model outputs and the correct outcomes can be used. There are measures of distance between the model output and the correct output, like the root mean square error (RMSE) or the mean absolute error (MSE), measures of goodness of fit, correlation, collinearity, ordering, or other aspects, e.g., the coefficient of determination r the q2 . The besRegression, as clustering, may be affected by outliers see .In order to find the best trade-off between model complexity and data availability, it is useful to train more than one model and compare their performances on the test dataset. In addition to model parameters most of the models have another set of hyper-parameters that need to be tuned to achieve optimal performances, like the number of neighbours in k-NN, the number of trees in a random forest or the number of layers, the number of units and the activation functions in a neural network.These hyper-parameters cannot be inferred directly from data like other training parameters, and need to be estimated by means of an explicit search in some parameter space. Care must be taken when performing hyperparameter optimization since models cannot be evaluated neither on the training set nor on the test set to avoid producing over-optimistic error estimates. The solution is to split the data into three datasets namely, training, validation and test sets . A commoWhen there is not enough data to ensure that model parameter estimates and performance estimates are stable, a simple split of the data is not the most efficient use of data. A more data-efficient approach is k-fold cross-validation, in which the dataset is randomly split into k subsets of approximately the same size, then iteratively, one of the k subsets is used as a validation set and the remaining k-1 subsets as training. The cross-validated estimate is then the average across the k runs, common choices for k are 5 or 10. The limit case where k is equal to the number of samples is called leave-one-out cross-validation.In case of classification, care must be taken during either type of splitting in ensuring that the class distribution is preserved across the splits. This is particularly important for the case of heavily imbalanced datasets, where some of the splits may completely miss the less represented classes. To overcome this problem, the split may be performed taking into account the class labels and ensuring that each, e.g., the validation set has the same proportion of samples from each class.Modeling of toxicogenomics data should always take into account the issues. For example, Minowa et al. identified genomic biomarkers for drug-induced renal injury from gene expression data, by applying filter based feature selection and linear classification algorithms. They evaluated their model by using a five-fold cross validation strategy and achieved high sensitivity and selectivity. The genes included in their model were primarily involved in DNA replication, cell cycle control, and oxidative stress and chemical stimuli.More recently, Furxhi et al. , compareIn the ML field particular attention has been given to DL methodology as a very good alternative for big data analytics with a high rate of success . This raAlgorithms for DL that have been used with success  include http://enalossuite.novamechanics.com) that is used by NanoSolveIT H2020 nanoinformatics project.The increasing interest in DL also favoured the creation of a high number of frameworks and platforms for the development of custom applications. Among those, the most used open source frameworks are Tensorflow, PyTorch and MXNet. Apart from frameworks, also platforms for the execution of computational experiments have become common. Many companies nowadays provide the hardware infrastructure required to train DL models as cloud instances in which is possible to develop “notebook” style general applications like Google Colab, Amazon Sagemaker and Azure Notebooks, as well as commercial platforms specialized on the analysis and integration of TGx data such as the Enalos Analytics , copy number variation (CNV), single nucleotide polymorphism (SNP) and protein-protein interactions (PPI). Each of these experimental data potentially provides complementary information about the whole studied organism ,164.Depending on the nature of the data and on the statistical problem to address, the integration of heterogeneous data can be performed at different levels: early, intermediate and late . Early iMoreover, different TGx studies integrated gene expression or RNA-Seq data with biological assays, clinical chemistry, therapeutic categories or molecular pathways to get increasingly exhaustive reasoning of biological mechanisms and cellular functions associated with adverse outcomes from environmental exposures and toxicants . For exaFurthermore, a different set of ML methods, both supervised and unsupervised have been proposed for multi-omic data analysis. For example, Pavlidis et al.  proposedFollowing the assumption that the relationship between structural properties and phenotypic effects of exposure is indirectly mediated by its MOA, an alternative approach to identify markers for toxicity could focus on defining hybrid predictive models that combine both structural properties and TGx features . For exaIn this third part of the three-article series review on transcriptomics data in TGx, we provided an overview of the state-of-the-art methodologies to analyze, interpret and model TGx data that are used to better explain the compounds’ MOAs and to perform toxicity predictions.The availability of open source transcriptomics datasets led to the development of different downstream analysis and modelling methodologies to answer specific research questions. For example, the BMD analysis allows identifying the minimal doses that affect the gene expression. The gene co-expression network analysis can elucidate the similarity/dissimilarity in treatment response. Read-across methods can be used to fill data-gaps and translate knowledge from existing compounds to the most similar ones. AOPs can be used to explain the links between a molecular initiating event after exposure and the final adverse outcome by creating a chain of relevant key events. Furthermore, ML methods can be used to create accurate and reliable models for toxicity prediction. Key aspects are robust and accurate predictions, rigorous model validation, well defined AD, and when possible an easy interpretation of model results. Predictive models that satisfy these requirements might assist the risk assessment and decision-making procedure .One major issue of concern is the reproducibility of data and the quality assurance that are of utmost importance for all data used for modelling. Lack of high-quality data will result in unreliable in silico models that will not be exploitable for regulatory purposes. A more thoughtful discussion about data generation can be found in the first part of this review series. Moreover, open data access, open protocols and publicly available meta-data annotations, although not the focus of this review, are integral for reproducible analyses as part of the FAIR Data Principles in order to make data findable, accessible, interoperable and reusable .In conclusion, TGx methodologies have a good potential to become part of the regulatory hazard assessment when all aspects from data generation to data preprocessing and modelling will be harmonized, and openly available for the scientific and regulatory communities. Furthermore, we believe that some methodologies and techniques implemented in other fields  could be translated in the contest of TGx. Eventually, future methods could combine ML algorithms and dose-dependencies methods in order to identify biomarkers of toxicity.This review can be considered the starting point to identify the best downstream analysis methodology to apply to TGx data depending on the problem in hand. It is important to highlight that each one of the described methods can be used individually, but they can also be concatenated in a pipeline to perform a more comprehensive TGx analysis. Moreover, it is important to note that all the modelling methodologies strongly rely on careful planning of the exposure conditions and robust data preprocessing, discussed in detail in the first and second parts of this review series.


The implementation of cancer precision medicine requires biomarkers or signatures for predicting prognosis and therapeutic benefits. Most of current efforts in this field are paying much more attention to predictive accuracy than to molecular mechanistic interpretability. Mechanism-driven strategy has recently emerged, aiming to build signatures with both predictive power and explanatory power. Driven by this strategy, we developed a robust gene dysregulation analysis framework with machine learning algorithms, which is capable of exploring gene dysregulations underlying carcinogenesis from high-dimensional data with cooperativity and synergy between regulators and several other transcriptional regulation rules taken into consideration. We then applied the framework to a colorectal cancer (CRC) cohort from The Cancer Genome Atlas. The identified CRC-related dysregulations significantly covered known carcinogenic processes and exhibited good prognostic effect. By choosing dysregulations with greedy strategy, we built a four-dysregulation (4-DysReg) signature, which has the capability of predicting prognosis and adjuvant chemotherapy benefit. 4-DysReg has the potential to explain carcinogenesis in terms of dysfunctional transcriptional regulation. These results demonstrate that our gene dysregulation analysis framework could be used to develop predictive signature with mechanistic interpretability for cancer precision medicine, and furthermore, elucidate the mechanisms of carcinogenesis. Biomarkers or signatures for predicting prognosis and therapeutic benefits are an indispensable part for implementing cancer precision medicine . The impThe most widely adopted strategies for identifying predictive signatures heavily depend on expression analysis of individual genes involving identification of differentially expressed genes (DEGs)  and geneAttributed to the crucial roles of gene regulation in fundamental cell processes, to build signatures with mechanistic interpretability requires studying relevant genes in the context of regulatory networks, that is, identifying specific regulators and their regulatory relationships that are dysfunctional in a given disease state . In the Since correlation analysis cannot distinguish direct associations from indirect associations , it has Boruta  on TCGA CRC dataset. Following the procedures in Materials and methods, we found that for all four types of cox models, C-indexes of 389 models fitted with the 389 dysregulations were significantly larger than those with 389 gene pairs randomly selected through four gene selection strategies , with the median ts <0.05 . We thents <0.05 , Exp_RFSConsidering that our identified 389 dysregulations could cover known carcinogenic processes  and exhiGiven that plasma membrane proteins show great potential as drug targets and diagnostic objectives, we narrowed down the 389 dysregulations to 126 plasma membrane protein-relevant dysregulations to build a prognostic signature with explanatory capability . StartinP-value = 6.52e−10; n = 107) and microsatellite stability (MSS) subgroup (n = 235) (Based on the expression data of the seven genes involved in 4-DysReg  and the n = 235) . Since Mn = 235) , it seemAt last, cross-validation on three datasets, TCGA CRC, GSE39582, and GSE17538, indicated that C-index, 1-, 3-, and 5-year survival AUC, and log-rank test of the testing sets maintained high levels compared to the training sets . Taken tWe compared the predictive accuracy of 4-DysReg for OS with previously reported CRC expression signatures, including RUNX3 , ColoPriADJC is preferred for curing CRC patients, and the guideline of ADJC is established on pathologic stage . Stage In = 321), which was used to calculate risk scores of samples with ADJC (n = 232) , with ADJC records ranging from stage II to stage IV, to explore the predictive power of 4-DysReg for chemotherapeutic benefit regardless of pathologic stage. An OS cox model was trained with the expression data of the seven genes involved in 4-DysReg on samples without ADJC (232) see .n = 84; HR = 0.380; 95% CI: 0.178‒0.813; n = 79; HR = 0.437; 95% CI: 0.181‒1.055; Furthermore, we analyzed the predictive capability of 4-DysReg for benefit of specific chemotherapy types including 5-FU and combined ADJC . Still, the samples with negative risk score basically showed better survival for both combined ADJC . GSE39582 contains 566 CRC tumor samples and 19 normal samples. GSE17538 contains 238 CRC tumor samples. Both datasets are based on GPL570 platform. The probes matching to multiple genes were deleted. When >1 probe could be mapped to the same gene, the probe with the highest value was taken to represent the expression level of the gene. The way of handling missing data was the same as that for TCGA data. The expression data were log2 transformed. Quantile method was adopted to normalize data between samples  and regarded as promoter regions. The promoter region of each gene was scanned with FIMO , fitted with (i) two genes’ expression data adjuvant with clinical information including age, gender, and pathological stage for OS, (ii) two genes’ expression data for OS, (iii) two genes’ expression data adjuvant with clinical information including age, gender, and pathological stage for RFS, and (iv) two genes’ expression data for RFS. For each cox model, C-index was calculated to measure the prediction accuracy. Meanwhile, four groups of controls were set with the following procedures, each containing the same number of randomly selected gene pairs as the identified gene dysregulations: (i) two genes were randomly selected from the gene list of preprocessed expression data; (ii) one DEG and one non-differentially expressed gene (non-DEG) were randomly selected from the gene list of preprocessed expression data; (iii) one regulation was randomly selected from the reference GRN; and (iv) one regulation whose target is DEG was randomly selected from the reference GRN. Still, four types of cox models were built for each random pair, and C-index was calculated for each model. At last, for each type of cox models, Wilcox test was used to check whether the C-indexes from the identified dysregulations were significantly larger than those from random pairs (one-way Wilcox test), i.e. whether the accuracy of cox models fitted by the identified dysregulations as a whole was higher than by controls. This process was repeated 100 times. The prognostic effect of the identified dysregulations was also validated in GSE39582 and GSE17538.For each gene dysregulation, four types of cox proportional hazard models were constructed in TCGA CRC dataset by using R package survival . The dysregulations involved in the final model were taken as prognostic signature.timeROC  survival analysis was then carried out to evaluate the difference in survival time between the two groups. The accuracy of the risk score for predicting survival was evaluated by C-index and AUC with timeROC . These aThe prognostic signature was further validated following a cross-validation procedure. That is, 60% of the samples were randomly selected as training set to fit a cox model as described above, and the left 40% were taken as testing set to calculate C-index, AUC at 1-, 3-, 5-year OS, and significance of KM survival curves. The cross-validation was repeated 100 times. This cross-validation procedure was respectively conducted in TCGA CRC and two independent datasets.P-value was <0.05.Pathway over-representation analysis was implemented on ConcensusPathDB , which iGSEA was performed with GSEA3.0 software . The genJournal of Molecular Cell Biology online.mjaa041_Supplementary_DataClick here for additional data file.


Exploring the molecular mechanisms of breast cancer is essential for the early prediction, diagnosis, and treatment of cancer patients. The large scale of data obtained from the high-throughput sequencing technology makes it difficult to identify the driver mutations and a minimal optimal set of genes that are critical to the classification of cancer. In this study, we propose a novel method without any prior information to identify mutated genes associated with breast cancer. For the somatic mutation data, it is processed to a mutated matrix, from which the mutation frequency of each gene can be obtained. By setting a reasonable threshold for the mutation frequency, a mutated gene set is filtered from the mutated matrix. For the gene expression data, it is used to generate the gene expression matrix, while the mutated gene set is mapped onto the matrix to construct a co-expression profile. In the stage of feature selection, we propose a staged feature selection algorithm, using fold change, false discovery rate to select differentially expressed genes, mutual information to remove the irrelevant and redundant features, and the embedded method based on gradient boosting decision tree with Bayesian optimization to obtain an optimal model. In the stage of evaluation, we propose a weighted metric to modify the traditional accuracy to solve the sample imbalance problem. We apply the proposed method to The Cancer Genome Atlas breast cancer data and identify a mutated gene set, among which the implicated genes are oncogenes or tumor suppressors previously reported to be associated with carcinogenesis. As a comparison with the integrative network, we also perform the optimal model on the individual gene expression and the gold standard PMA50. The results show that the integrative network outperforms the gene expression and PMA50 in the average of most metrics, which indicate the effectiveness of our proposed method by integrating multiple data sources, and can discover the associated mutated genes in breast cancer. Breast cancer is considered to be the most prevalent cancer among women and the second common cause of death in both developed and undeveloped countries. It is caused by multiple factors including genomic, transcriptomic, and epigenomic involvement in its formation and development. With the development of technology, understanding the pathogenesis of cancer from the perspective of molecular contributes to effective diagnosis and treatment. The large-scale cancer genomics project, The Cancer Genome Atlas (TCGA) , has proIn general, the cancer transcriptome contains gene expression, including messenger RNA (mRNA), long non-coding RNA (lncRNA), and microRNA (miRNA). Previous studies focused on utilizing the gene expression profile to successfully diagnose individuals based on the differential gene expression  and otheOn the other hand, studies have shown that somatic mutations frequently perturb the expression level of affected genes and thus disrupt the pathways controlling normal growth . For exap-value and the adjusted p-value to correct for multiple-hypothesis testing  and thus obtain the differential expression network. Meanwhile, an integrative network is constructed by combining the mutation network and the differential expression network. Thirdly, we rank the genes in the integrative network by mutual information (MI) and select the top 50 genes, which are highly correlated with breast cancer. Finally, we use the Bayesian optimization method to optimize the classification model, gradient boosting decision tree (GBDT), which is further applied to assess the features selected from the previous step. In terms of evaluation metrics, the traditional metric of accuracy does not consider the sample imbalance, so we propose a simple and effective metric, balanced accuracy, to reveal the ability of the different model to classify positive and negative samples.As the molecular network has been verified to be effective for the biological discovery of cancers, current studies utilized the network across different types of cancer or cancer subtypes. However, the objective of most researches is the universality of the methods, which makes it difficult to be equally effective in all disease types. In this study, we aim to construct an efficient method of architecture for the diagnosis of breast cancer based on the network of somatic mutation and gene expression. We are focused not only on finding more biomarkers but also on the classification performance of the model. First, the somatic mutation is used to generate a binary mutation network; similarly, an expression network is obtained from the gene expression profiles. Then, for the expression network, we compute both the observed We used statistical and machine learning methods to develop this novel method for feature selection and classification, including the preprocessing of data, filter method, and embedded method for feature selection, processing of imbalanced data, and the final classification model. In this research, we use publicly available breast cancer datasets (BRCA) from TCGA, including transcriptome gene expression and somatic mutation. Considering the different structures of these two types of data, we used different methods to preprocess them. The BRCA gene expression dataset comprises 1222 samples and 57,063 genes. There are 113 normal samples and 1109 tumor samples. We used the edgeR package to filter the genes expressed in small amounts in most samples and normalized the data. The gene expression data was reduced from 57,063 to 34,465 by deleting the genes expressed in small amounts in most samples.1, s2, …sn}, n is the total number of samples, and si represents the sample i. Gene set G = {g1, g2, …gm}, m is the total number of mutation genes, and gj represents the gene j. In the set of sample number C = {c1, c2, …cm}, ck represents the number of samples with “1” in gene k. The set C can be calculated by the number of “1” in each row in The somatic mutation data comes from the simple nucleotide variation (SNV) in the TCGA-BRCA project. The data file includes SNP, INS, and DEL, three types of mutations. The important fields in the data file are Hugo_Symbol (gene name), Variant_Type, and Tumor_Sample_Barcode (sample name). Statistically, the somatic mutation data contains 18,127 genes and 986 samples. To get the mutation frequency of each gene in all samples, we use a Perl script to process the data file. For example, if gene A is present in sample S, that means sample S has a mutation in gene A, then we code it as “1,” otherwise we code it as “0.” According to Somatic mutations in cancer genomes frequently perturb the expression level of affected genes. Then, the pathways controlling normal growth are disrupted . SimilarOur study is based on the assumption that mutations may cause changes in the cell’s state, such as underexpression or overexpression of different genes. Then, we combine the somatic mutation network with the gene expression network to obtain an integrative network. In the integrative process of the two types of networks, we refer to the gene expression network to obtain the expression value of the somatic mutation genes in the mutation network. It is important to note that in the subsequent classification task, the normal samples in the expression network are added as the control group.i and sample j in cancer samples and normal samples; FC is defined as:Fold change (FC) is used to calculate the differential multiples of gene expression values between cancer samples and normal samples, which is the basic method for detecting differential genes, and represents the expression values of feature When FC exceeds the initial set threshold, it can be considered that the feature is different, and it is generally considered that there is a significant difference when the difference multiple is more than 2. FC can directly obtain the differentially expressed values, but in the absence of false-positive control, the rate of false-positive results is relatively high.According to statistical theory, in multiple-hypothesis testing, it is important to control the probability of making mistakes in multiple statistical inferences, called FDR. FDR can be used to analyze deferentially expressed genes to control the proportion of false positives . Table 2q. In general, keep FDR below 0.01, or ensure that there is at most one false positive for every 100 positive hypotheses. Feature genes with significant differences can be identified by FC and FDR, but these two methods do not evaluate the classification performance of these features.The number of false positives in multiple-hypothesis tests can be controlled by controlling that FDR is below the threshold Fold change and FDR are applied to integrative data to select the differentially expressed genes. By comparing the classification balanced accuracy under different FC and FDR thresholds shown in Gi, and the label column is defined as L. MI is represented as the MI between the gene Gi and the label L. The calculation equation is Eq. 3.Mutual information  is a useH(Gi) is the information entropy of the gene column Gi, H(L) is the information entropy of the label L, and H is the joint information entropy of Gi and L. According to information theory, the information entropy is a measure of the uncertainty of a random variable. Suppose X is a random variable, and the range of possible values is xS, x ∈ Sx and the probability is p(x); the information entropy of X is defined as:H is the joint information entropy, defined as:p is the joint probability density function. MI can be calculated according to Eqs 4 and 5. In our study, MI is used to measure the dependency between a feature and the classification type. In general, the greater value of MI indicates that the feature contains more information for classification. Therefore, we rank the MI values of each feature and selected the top M features from the integrative data, respectively. The final objective of this method is to remove irrelevant features to reduce the dimension of integrative data. We set different values of M to compare the classification-balanced accuracy and obtain the best value of M. The result in The filter methods obtain a feature subset for which the discriminative capability is limited for classification purposes. Embedded methods can be used to search the optimal feature subset by a given classifier. In the training procedure, the features with high importance can be selected by ranking and the classification algorithm is optimized simultaneously. It is helpful to build a strong link between the feature subset and the classifier. The GBDT is an ensemble learning algorithm based on GBM, which is proposed by Friedman . During The boosting tree  is an acHowever, it is tedious and important work to tune the hyperparameters when conducting the GBDT, because it greatly affects the performance of the algorithm. Manual tuning is time-consuming; grid and random searches  require f(X). However, machine learning just likes a black box, which means we only know the input and output; f is hard to be sure. So we should turn our attention to a function that can be solved. Assume function, we need to find in:Suppose hyperparameters set , the relationship between this set, and the loss function that need to be optimized, defined as Here, we chose Hyperopt in Python library, which adopted Tree Parzen Estimator (TPE), which used the Gaussian Mixture Model  to learnClass imbalance is a situation in which the number of training samples of different categories varies greatly in the classification task. There are many strategies to deal with the imbalance problem, such as undersampling and oversampling. EasyEnsemble is a method of undersampling, proposed by N and T denote the sample set of normal class and that of tumor class, respectively. clf in the validation set, respectively. These two weights represent the different capacities of the given classifier for different types of samples. In the final testing stage, the optimized GBDT is conducted as the classifier to predict the independent test set; accn and acct. So the final balanced accuracy is defined as:Let The core procedure of calculating the weighted metric for balanced accuracy is described in The following metrics are used to evaluate the performance of the classification model in this study:Accuracy: Sensitivity: Specificity: Precision: F1 score: In this study, the tumor sample is positive, and the normal sample is the negative sample, where TP (true positive) is the number of tumor samples predicted as tumor, FP  is the number of tumor samples predicted as normal, TN (true negative) is the number of tumor samples normal and predicted as normal, and FN  is the number of normal samples and predicted as tumor. Meanwhile, the AUC is obtained.Due to that the number of samples is much smaller than that of the features, in this study, first, we split the dataset into 10 equal-sized datasets. Then, we divide the datasets into 80% learning set and 20% test set and ensure that the test set does not participate in any training process . FinallyIn our experiments, the training set is used to train the classifier. The obtained parameters are verified on the validation set. In addition, we calculate log(FC)| > 1.0(1) FC: |FDR 0.05(2) FDR: (3) MI: select the top 50 features of MI value ranking(4) Bayesian optimization: tuning the parameters of GBDT with Bayesian optimization using the 50 features to get the optimal model.These methods are combined in the ways shown in Case 1: None of the above methods are used.Case 2: FC and FDR are used to obtain the differentially expressed genes.Case 3: FC + FDR, MI are used to select informative features.Case 4: FC + FDR, MI, and Bayesian optimization are adopted to optimize GBDT, and this case is the proposed method.The testing accuracy is obtained by the classifier GBDT on the independent test set. The results shown in case 1 are the classification accuracy using GBDT without any feature selection. It can be observed that the GBDT without any feature selection obtains a testing accuracy of about 97.96%, but the testing balanced accuracy is only about 85.47%, which implied the learning efficiency of the GBDT without feature selection is not much high. In case 2, although FC and FDR effectively reduce the running time, it does not improve the accuracy significantly, because they ignore the correlation between features. In case 3, we add MI to further select key features, and the results show that there is an improvement (1.58%) in balance accuracy and a significant reduction in running time. In case 4, we use Bayesian optimization to optimize GBDT to obtain the optimal model. According to the results, we conclude that the accuracy and balanced accuracy are improved by 0.74 and 5.14%, which were compared with case 3. Particularly, the proposed method shown in case 4 obtains the highest testing accuracy and balanced accuracy. The performance of testing balanced accuracy is improved by 13.85%, compared with the method in case 1. From the perspective of vertical comparison, the features selected by the proposed method have better classification performance. From the perspective of horizontal comparison, balanced accuracy improves more than traditional accuracy, which indicates that the proposed model shows greater advantages when the sample balance is considered.Bayesian optimization aims to find the minimum value of the objective function by establishing a proxy function (probabilistic model). The proxy function is easier to optimize than the objective function , so the In the comparative experiments, we select three other classifiers, SVM, KNN, and RF. (1)SVM: C = 1, kernel = “linear”(2)KNN: n_neighbor = 7, metric = “manhattan”(3)RF: max_depth: 46, min_sample_leaf: 2, min_sample_split: 94, n_estimators: 75To explore the effect of the integrative dataset, we apply the proposed method to individual gene expression and integrative dataset, respectively. Besides, we choose PMA50 as the control model. PMA50 refers to a set of 50 genes selected by The results in IQGAP3 (KIF4A (TSHZ2 (MKI67 (TNXB (KIFC1 (KDM5B (PPEF1 (RYR3 (TMEM132C (FANCD2 (ATAD2 (KIF26B (BRCA2 (BLM (ARFGEF (The 50 genes (listed in the IQGAP3 , KIF4A (3 (KIF4A , TSHZ2 (A (TSHZ2 , MKI67 (2 (MKI67 , TNXB (H67 (TNXB , KIFC1 (B (KIFC1 , KDM5B (1 (KDM5B , PPEF1 (B (PPEF1 , RYR3 (SF1 (RYR3 , TMEM132TMEM132C , FANCD2  (FANCD2 , ATAD2 (2 (ATAD2 , KIF26B  (KIF26B , BRCA2 (B (BRCA2 , BLM (ArCA2 (BLM , and ARF (ARFGEF , which aBRCA2, KDM5B, and IQGAP3 are associated with mammary gland epithelial cell proliferation and gland development; BLM, BRCA2, CENPE, CENPF, KIFC1, CKAP, CIT, TTC28, KIF4A, and ASPM are associated with cell division; BRCA, CENPE, CENPF, FANCD2, KIFC1, MKI67, KIF4A, and ASPM are associated with organelle fission; BLM, BRCA2, CENPE, CENPF, EGFR, FANCD2, MKI67, CKAP5, and TTC28 are associated with regulation of the mitotic cell cycle; ABCA10, ABCA9, ABCA8, and ABCA6 enrich in the pathway of ABC transporters; and EGFR, FN1, RELN, and TNXB enrich in the pathway of human papillomavirus infection. The main GO and pathway are shown in Gene Ontology and pathway analysis produces biological function and pathway enriched for mutation genes. The result reveals that This research presents a Staged Feature Selection method for breast cancer classification based on gene expression and somatic mutation datasets. In the proposed method, FC and FDR were used to select differentially expressed genes, MI was adopted to remove the irrelevant and redundant features, and an embedded method based on GBDT with Bayesian optimization was presented to obtain the informative features. Besides, the weighted metric was proposed to evaluate the classification accuracy, which could avoid the impact of sample imbalance on classification. The experiment results showed that the proposed method selected 50 feature genes and achieved the accuracy of 99.51%, the balanced accuracy of 97.31% and the sensitivity of 99.64%, the specificity of 98.26%, the precision of 99.82%, the F1 score of above 99.73%, and the AUC of 98.95%, which was superior to the other three classifiers. It was verified that the proposed method was an efficient tool for feature selection in breast cancer classification.The results presented the effectiveness of integration with gene expression and somatic mutation data for breast cancer classification, which indicated that it could provide more useful information for cancer classification by integrating multiple information. However, this study only focused on breast cancer, and the scalability of the proposed method on other types of cancers remained to be further explored, which will provide helpful information for cancer prevention and treatment. Therefore, in future work, we will apply the approach to classify other types of cancer, explore ways to incorporate more relevant data, and introduce other techniques to boost our method. Besides, the pathogenesis of some biomarkers discovered by the proposed model still has to be verified by biological experiments.https://portal.gdc.cancer.gov/repository.Publicly available datasets were analyzed in this study. This data can be found here: QJ processed the data, designed the algorithm and the programming codes, and wrote the manuscript. MJ supervised the project and revised the manuscript. Both authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Cancer of unknown primary (CUP), representing approximately 3-5% of all malignancies, is defined as metastatic cancer where a primary site of origin cannot be found despite a standard diagnostic workup. Because knowledge of a patient's primary cancer remains fundamental to their treatment, CUP patients are significantly disadvantaged and most have a poor survival outcome. Developing robust and accessible diagnostic methods for resolving cancer tissue of origin, therefore, has significant value for CUP patients.We developed an RNA-based classifier called CUP-AI-Dx that utilizes a 1D Inception convolutional neural network (1D-Inception) model to infer a tumor's primary tissue of origin. CUP-AI-Dx was trained using the transcriptional profiles of 18,217 primary tumours representing 32 cancer types from The Cancer Genome Atlas project (TCGA) and International Cancer Genome Consortium (ICGC). Gene expression data was ordered by gene chromosomal coordinates as input to the 1D-CNN model, and the model utilizes multiple convolutional kernels with different configurations simultaneously to improve generality. The model was optimized through extensive hyperparameter tuning, including different max-pooling layers and dropout settings. For 11 tumour types, we also developed a random forest model that can classify the tumour's molecular subtype according to prior TCGA studies. The optimised CUP-AI-Dx tissue of origin classifier was tested on 394 metastatic samples from 11 tumour types from TCGA and 92 formalin-fixed paraffin-embedded (FFPE) samples representing 18 cancer types from two clinical laboratories. The CUP-AI-Dx molecular subtype was also independently tested on independent ovarian and breast cancer microarray datasetsCUP-AI-Dx identifies the primary site with an overall top-1-accuracy of 98.54% in cross-validation and 96.70% on a test dataset. When applied to two independent clinical-grade RNA-seq datasets generated from two different institutes from the US and Australia, our model predicted the primary site with a top-1-accuracy of 86.96% and 72.46% respectively.The CUP-AI-Dx predicts tumour primary site and molecular subtype with high accuracy and therefore can be used to assist the diagnostic work-up of cancers of unknown primary or uncertain origin using a common and accessible genomics platform.NIH R35 GM133562, NCI P30 CA034196, Victorian Cancer Agency Australia. Cancer of unknown primary (CUP) is defined by the presence of metastatic disease with no identified primary tumour despite extensive clinical and histopathological investigations In this study, we designed a deep neural network model to identify a tumour's primary site of origin using RNA-seq gene expression across 32 cancer types available data from the two largest pan-cancer genome consortia, The Cancer Genome Atlas (TCGA) and International Cancer Genome Consortium (ICGC). A machine learning classifier was also developed to discriminate the molecular subtypes of 11 cancer types once the primary site of origin is predicted. The classifier can be applied to clinical samples using widely accessible RNA-seq protocols.Metastatic tumours retain the gene expression profile of the primary tissue of origin enabling the development of machine learning methods for predicting tissue of origin. The development of a gene-expression tissue of origin test using the widely available RNA-seq method enables democratization of this method to other diagnostic labs. The tissue of origin classification concept can also be extended to incorporate molecular subtyping of cancer into a single test, which provides additional information to the workup of cancer patients.Alt-text: Unlabelled box1The vast majority of contemporary cancer treatments, including targeted therapies, are still applied with knowledge of the patient's primary tumour. However, 3-5% of all cancer patients have metastatic tumours where routine testing cannot locate the primary site resulting in a diagnosis of cancer of unknown primary (CUP) It is known that most metastatic tumours harbour a cellular phenotype that resembles their original tissue of origin. Immunohistochemistry (IHC) is commonly used in the diagnostic workup of metastatic cancers by using antibodies targeting protein antigens that have restricted cell type staining patterns. However, the utility of IHC-based classification is limited to small antibody panels and these are not standardized between laboratories. Furthermore, the application of multiple immunostains can rapidly consume often limited amounts of tumour tissue that are increasingly also required to perform other analyses, including DNA and RNA sequencing. Several studies have used molecular profiling including gene-expression (mRNA or miRNA) and DNA methylation profiling to predict CUP tissue of origin, and some of these tests have been commercialized . Like foRNA and DNA-based tissue of origin classifier methods previously reported have shown high and comparable performance. The most extensively validated RNA-based tissue of origin test is a commercial 92-gene real-time PCR based test (CancerTypeID), which reports a classification accuracy of 87% for primary and 82% in for metastatic tumours using a large validation series Looking beyond tumour type prediction, molecular subtype identification—with or without the primary site identification—may also lead to enhanced therapeutic options for CUP patients. The Cancer Genome Atlas (TCGA) Research Network and International Cancer Genome Consortium (ICGC) studies have shown that cancers with a known primary site can be further classified into molecular subtypes with distinct clinical outcomes and therapeutic options  and thatHere we introduce the 1D-Inception model, a machine learning framework to predict the primary site and molecular subtype of cancer samples based on the classification of gene expression data . The pri22.12.1.1. The gene expression matrix of each cancer was converted into a Biobase ExpressionSet object TCGA gene expression data: TPM (transcripts per million) ICGC gene expression data: TPM (transcripts per million) After combining TCGA and ICGC datasets, we have the gene expression data matrix with 18,217 samples spanning 32 different tumour types and 20,531 genes as the training dataset  in each class (cancer type) as determined by the difference between the median expression of each gene in the in-class sample relative to the out-of-class samples (p < 0.001). Median expression was used (instead of the mean) due to its robustness to extreme values. For ResNet, we similarly selected the 70 most DEGs in each class to meet the input size requirement. DEGs may be overlapping between different classes. The DEGs sets from each class were combined, merged, and used in training each model, from which 791 and 1024 unique “feature genes” common to all external validation sets were selected for the 1D-Inception/1D-CNN and ResNet, respectively. The genes were ordered according to their chromosomal locations. In parallel, we selected 241 genes by picking the 10 most differentially expressed genes in each class (cancer type) to observe the performance of each model with a small feature set size. With the combined training dataset from TCGA and ICGC, we selected the 40 most differentially expressed genes (DEGs) in each class (cancer type) as determined by the difference between the median expression of each gene in the in-class sample relative to the out-of-class samples (p < 0.001), and combine the overlapping of different DEGs sets. Finally, 817 unique “feature” genes were selected for the 1D-Inception model construction (See Table S6 for the 817 Entrez Gene IDs).https://gdc.cancer.gov/resources-tcga-users/tcga-code-tables/sample-type-codes)) for 11 cancer types as follows (using the TCGA study abbreviations): breast invasive carcinoma (BRCA); cervical squamous cell carcinoma and endocervical adenocarcinoma (CESC); colon adenocarcinoma (COAD) and rectum adenocarcinoma (READ), which we combined into a single cohort (COADREAD); esophageal carcinoma (ESCA); head and neck squamous cell carcinoma (HNSC); pancreatic adenocarcinoma (PAAD); pheochromocytoma and paraganglioma (PCPG); prostate adenocarcinoma (PRAD); sarcoma (SARC); skin cutaneous melanoma (SKCM); and thyroid carcinoma (THCA). The metastatic gene expression matrix consisted of 394 samples spanning 11 cancer types and 16,383 genes.TCGA metastatic data: To validate the primary tumour type predictor accuracy, we utilized TCGA metastatic samples (sample type code “06” (2.2https://bcbio-nextgen.readthedocs.io/). Raw gene expression counts were obtained from length scaled transcripts per million (TPMs) using the tximport R package version 1.12.0 Clinical validation samples included ninety-three tumours processed at two sites in the USA and Australia. Twenty-three formalin-fixed paraffin-embedded (FFPE) specimens samples representing 6 cancer types were obtained from clinical testing over 4 years in the JAX CLIA lab. Seventy metastatic FFPE tumours representing 18 cancer types were profiled at the University of Melbourne (UOM). Both sample cohorts were subject to RNA-Seq with double-blinded treatment (only tissue of origin and diagnosis were known) for clinical validation of our 1D-Inception model. All FFPE specimens were macro dissection-enriched for extraction and total RNA purified using either the Qiagen AllPrep DNA/RNA FFPE Kit or RNA FFPE Kit . At JAX CLIA lab 50 ng of RNA was subjected to sequencing using the KAPA RNA PyperPrep Kit with RiboErase (HMR) protocol and sequencing by synthesis on an Illumina NextSeq 500 instrument. At the University of Melbourne Centre for Cancer Research RNA-seq libraries were prepared using the NEB-next NEBNext Ultra II Directional RNA Library Prep Kit for Illumina® and libraries sequenced on the Illumina Nova-Seq 6000. Raw BCL files generated by the sequencer were converted to FASTQ files using CASAVA. The RNA-Seq data was aligned under to the human transcriptome version hg38 using kallisto version 0.46.0 2.3The expression data was scaled for each patient sample independently for data normalization, i.e. the expression data were normalized by subtracting the mean and dividing by the square root of the variance of gene expression from the same patient.All expression data were log2-transformed. After filtering, the genes in each dataset were scaled to zero mean expression and unit variance for each patient. This scaling allows expression to be measured in terms of standard deviations and affords platform-independent use of subsequently trained models.2.4To predict primary tumour types, we developed a 1D Inception model and compare it with two other deep learning models: ResNet and 1D-CNN on the TCGA dataset. Performance metrics and contingency table for all primary site predictors in cross-validation on TCGA dataset and metastasis validation are listed in Table S1-S2. The Talos hyperparameter space for each model is listed in Supplementary Text.2.5Due to limited samples, we chose the ResNet V50 architecture 2.5.11D-CNN is a good model candidate since the 1D filters learned can detect different spatial shapes in the expression matrix. Thus, we strictly ordered our 791 features according to the chromosomal location. Our experiments with 1D-CNN utilized a 1D convolutional layer followed by one max-pooling layer. Dynamic fully connected (FC) layers with ReLu activation functions were built with Talos The 1D-CNN model was constructed using the Keras framework (v2.2.4) 2.5.2The 1D-Inception network is used to enhance the 1D-CNN network by considering multiple 1D convolutional kernels with different sizes at the same time. We chose this architecture because the 1D-Inception model was found to have superior performance on image data sets 2.6The schema for predictor design for primary site classification is depicted in All CNN models, 1D-Inception, 1D-CNN, and ResNet were trained using the same feature selection and cross-validation schedule on the TCGA dataset. Each model was then trained using a 10-fold cross-validation procedure as follows. The expression set was partitioned into 10 random subsamples and for each partition: 2.7The 1D-Inception architecture for TCGA and ICGC dataset is shown in Because this network allows the three convolutional modules to have different max-pooling layer window sizes and dropout layer keep-probabilities, we can utilize as few as 817 genes in the model to achieve a high overall top-1-accuracy of 98.54% in cross-validation and overall top-5-accuracy of 99.94%. This outperforms alternative methods that usually consider the whole gene set. This is because the lower-dimensional data has fewer patterns than the high dimensional data, thus it requires more kernel size options to detect the limited number of patterns. A third strength of this architecture is that the redundant layers can be reduced through Talos performance checking. As shown in 33.1Molecular subtype information was downloaded from cBioPortal ,52 for 3For each of the 11 primary tumour types with established molecular subtypes, a model was constructed as described above using the scaled, log2-transformed expression of the sample corresponding to the selected primary type as input. For each cancer type, features were selected by computing the differential gene expression (p<0.001) in each subtype in comparison with the other subtypes of the same cancer type. The list of gene features is shown in Table S7.https://www.ebi.ac.uk/ega), contains 1,784 breast cancer samples For the external validation of our subtype predictors, we acquired two additional microarray datasets. The first, accession number GSE9899, contains 215 ovarian cancer samples 3.3randomForest (version 4.6-14) for the training and testing and caret (version 6.0-79) for tool development. Unless otherwise specified, default parameters were chosen for model construction. Performance metrics for subtype predictor and pan-cancer predictor are listed in Table S3-S5. Comparable results were obtained using XGBoost, another ensemble model (Supplementary tables S9) but Random Forest is chosen as the default subtyping algorithm because of the superior performance in the external validation set.We evaluated several popular machine learning algorithms to develop predictors for molecular subtype identification and chose random forest (RF) model using R-packages 3.3.1The random forest algorithm employs a collection of decision trees constructed from bootstrapped input data and classification is done by majority voting among the ensemble of trees 3.4The Jackson Laboratory (JAX) Institutional Review Board (IRB) has reviewed theDetermination of Human Subjects Research form for the project indicated above and has determined that this project does not meet the definition of human subjects research under Laboratory policy and applicable Federal Regulations.This determination is based on the fact that the genomic summary data from RNA-seq assays is provided with no individual-level genomic data shared in publication. RNA-seq analysis of patient samples at the University of Melbourne was done under an approved protocol by the human research ethics committee at the Peter MacCallum Cancer Centre (Protocol: 11/117).3.5Each classification algorithm (predictor) was compared using overall accuracy, per class accuracy, sensitivity, and specificity. For datasets with class-imbalance, reporting accuracy alone is a misleading metric to gauge performance. For example, more than 50% of the training data from the TCGA network consists of tumours from only 9 of 32 classes, with BRCA making up 12% of the training dataset, and tumour classes are similarly imbalanced in the external validation datasets. Instead, we report classification performance as a combination of overall and per-class performance. Per class metrics are computed using a one-versus-all scheme. TP = true positive, TN = true negative, FP = false positive, FN = false negative: Metrics of performance are calculated as follows:Per class:Additional metrics such as specificity, sensitivity , and F1 score are included in supplementary tables S1-S5, S8.44.1Methods); it is important to note that the training data contained no metastatic tumour samples. The filtered cancer cohorts  are shown in RNA-seq cancer data was sourced from the TCGA and ICGC compendium datasets. To generate a classification training dataset, we combined COAD and READ into one single cohort (COADREAD) based on TCGA consortium findings To gauge the performance of our 1D-Inception model, we first compared its performance to that of two other popular CNN models, the 1D convolutional neural network (1D-CNN) model and Residual Net (ResNet) To further improve our 1D-Inception performance, we combine the TCGA and ICGC datasets as a combined dataset to redo the training process and updated the architecture b. The 1DThe major misclassifications observed during cross-validation were primarily within organ systems a. Of notMethods) and coloured by primary tumour type from the TCGA dataset  and pan-gynecological performing the worst with 60.4% accuracy and 62.4% median sensitivity. The pan-gynecological cluster with the worst performance, subtype C2 with only 41% sensitivity, was almost solely differentiated from subtype C1 by hypermutation patterns 4.4To further validate the cancer subtype predictors, we classified samples from two external datasets: ovarian cancer (GSE9899) The ovarian cancer subtype predictor d attaine5It is widely appreciated that cancer is a disease at the scale of the entire genome, but it remains difficult to effectively translate this complexity into clinical utility, especially in CUP. An important piece of information that is relevant for clinical care in all cancer settings is knowledge of primary tissue of origin. It can, therefore, be presumed that identifying tissue of origin and perhaps even the molecular subtype of a CUP tumour is required to guide optimal treatment The performance of our classifier was similar to that reported in other studies achieving an accuracy of 75.7%. Importantly, in testing CUP-AI-Dx we report the accuracy across a balanced representation of tumour types in an independent clinical series and report the accuracy of all cases tested regardless of prediction confidence score. Although CUP-AI-Dx performs well for most cancers, some tumour types showed a drop in classification accuracy. Classification can be challenging among tumours of the same organ system such as gastrointestinal tumours and also among tumours with similar histology, such as squamous cell carcinoma. Classification inaccuracies were noted in our clinical series among pancreatic adenocarcinoma and cholangiocarcinomas which is not surprising given that in clinical practice this distinction cannot be made with certainty in many cases Although our study performs well on identifying most solid cancers, it may not account for all biological differences observed among tumours or adequately represent rare cancers. The tissue of origin diagnostics, whether IHC-based or using molecular profiling, makes the fundamental assumption that metastases, including CUPs, retain features of the primary cell or tissue of origin. Although in this study we did not test CUP tumours, previous studies have shown that latent primary CUP, where a primary becomes known in time, can be predicted with similar accuracy to metastatic tumours of known origin ,37, whicThe path to improving the tumour type classification accuracy may be to consider including other potential features such as somatic point mutations In summary, we have demonstrated the utility of our machine learning algorithms to decode gene expression profiles and better meet the clinical challenge of identifying the primary site and the molecular subtype of multiple cancers. These predictors, including the deep learning-based predictor, will be made available as open-source software, freely available for academic non-commercial use. To make these tools available to as wide an audience as possible, we offer our models and results in a publicly available software package, which can be applied to other datasets to reproduce the results presented here.