After its introduction in 1982, the Hopfield model has been extensively applied for classification and pattern recognition. Recently, its great potential in gene expression patterns retrieval has also been shown. Following this line, we develop Hope4Genes a single-sample class prediction algorithm based on a Hopfield-like model. Differently from previous works, we here tested the performances of the algorithm for class prediction, a task of fundamental importance for precision medicine and therapeutic decision-making. Hope4Genes proved better performances than the state-of-art methodologies in the field independently of the size of the input dataset, its profiling platform, the number of classes and the typical class-imbalance present in biological data. Our results provide encoraging evidence that the Hopfield model, together with the use of its energy for the estimation of the false discoveries, is a particularly promising tool for precision medicine. From its introduction in 1982, the Hopfield model has been widely applied in computer science as associative memory to store a number of patterns and retrieve them from noisy/corrupted inputs. Its application for classification and pattern recognition spans numerous fields of research, from economy to medicine and social sciences3. In the past few years, thanks to the increasing availability of high-throughput data, the Hopfield network has been also applied to describe the emergence of gene expression patterns5 or, even more ambitiously, to model tissue differentiation8 and to estimate the pseudotime from single-cell RNA-seq data12. These last studies proved that the memory retrieval ability, characteristic of the Hopfield model, has a great potential for the analysis of transcriptomic data.The Hopfield model is the most popular and paradigmatic example of attractor neural network, i.e. a network whose spins evolve towards stored attractor patterns13. The five molecular subtypes of breast cancer , identified and intensively studied in the last 10 years, represent a typical example15. Multiple approaches have been proposed for cancer subtyping, most of them based on: matrix factorization16, clustering17, networks18 and deep learning20. All these algorithms are aimed at performing an unsupervised classification of a set of patients and at identifying a gene signature correlated with the identified classes. Once the subtypes have been identified, their usage for therapeutic decision-making on a new patient, requires to solve a class prediction problem. Class prediction requires in input a signature, i.e. a set of genes coherently over- or under-expressed in patients with a certain phenotype of interest, and it is aimed at associating each sample of the input dataset to a class (e.g. poor vs good prognosis) based on the coordinate behavior of the genes belonging to the signature.One of the major issues of personalized medicine is the classification of patients affected by a given disease into groups homogeneous with respect to molecular, prognostic or therapeutic features23. Consequently to the spread in gene signatures production, many class prediction algorithms have been introduced. Among them Nearest Template Prediction (NTP), based on associating a sample to the class centroid at lower distance, was found to outperform previous methods and it represents the state-of-art in the field24. The main advantages of NTP with respect to previously designed algorithms, such as those based on supervised learning approaches19, are represented by: (i) the absence of a training phase and the consequent risk of overfitting; (ii) the possibility to classify one single sample per time, fundamental for an effective implementation of the tool in the clinics and (iii) the implementation of a False Discovery Rate (FDR) to asses the prediction confidence.In the last years, transcriptomic signatures able to differentiate distinct subtypes of tumors, to identify important cellular responses, to predict clinical outcomes and to model the activation of signaling pathways have been extensively developedDue to the complexity of cancer, the high-dimensionality of trascriptomic data and the class-imbalance typical of these classification problems, the overall performances of most current methods still need to be further improved and more accurate and robust methods need to be developed.https://github.com/lcan88/Hope4Genes, preserves the three advantages implemented in NTP . Moreover, it associates to each sample a secondary class, useful to better characterize those 5–10% of samples that generally result to be unclassified by existing algorithms. Once tested on the same data to which NTP was originally applied, Hope4Genes obtained competing results, proving that the application of the Hopfield model for class prediction of transcriptomic data, interesting in itself, has also a great potential in precision medicine.Inspired by the promising results obtained by the Hopfiled model in the analysis transcriptomic data, we here developed Hope4Genes a new algorithm for class prediction based on a Hopfield-like model. Hope4Genes, available at https://github.com/lcan88/Hope4Genes, a single-sample class prediction algorithm for transcriptomic data  gene signatures are typically mutually non-overlapping partitions of the gene set, thus class templates are strongly correlated and (ii) gene signatures are usually strongly unbalanced, in fact their sizes usually show a wide variability, with classes represented by hundreds of genes and classes composed by few dozens of genes. The consequence of (i) is that the Hopfield model can converge to spurious memories. To overcome this problem, we modified the classical model proposing an asymmetric, diluted version of the Hopfield model (see Methods). To face problem (ii), we performed a signature reduction combined with a resampling procedure and we analytically derived the optimal threshold (Γ) for the size of the signature reduction (see Methods). For these reasons we refer to the algorithm here proposed as a Hopfield-like model. An example of the final structure of the model is reported in Fig. 24, representing the current state-of-art in the field of class prediction for transcriptomic data26. We considered in particular:Example 1: Prediction of Acute Lymophoblastic Leukemia (ALL) and Acute Myeloid Leukemia (AML). 35 leukemia samples to be classified into AML and ALL based on the gene signature defined in27.Example 2: Cross-platform prediction of Estrogen Receptor (ER) positivity in breast cancer. 49 breast cancer samples from28 to be classified according to the gene signature obtained in24.Example 3: Cross-species prediction of liver cirrhosis between human and rat. 12 liver cirrhosis rat samples are classified through a signature generated on human samples affected by the same pathology in29.Example 4: Prediction of multiple tissue types. A dataset containing 52 samples collected from four different tissues: breast, prostate, lung and colon is classified based on the signature defined in30.Example 5: Prediction of breast cancer molecular subtypes: Basal-like, HER2, Luminal A, Luminal B, and Normal. This last example is the one representing a real-world clinical application. To study how the preformances of the algorithm are influenced by the size of the input dataset, its profiling platform and the typical class-imbalance present in biological data we divided the example in four subcases obtained by varying the transcriptomic dataset and the considered subtypes (from 3 to 5):Example 5.1: a dataset of 198 samples31 is classified into 5 subtypes: Luminal A, Luminal B, HER2, Basal and Normal.Example 5.2: we classified a dataset of 286 samples32 in 4 subtypes: Luminal A, Luminal B, HER2 and Basal.Example 5.3: 53 samples of 33 are classified into 3 subtypes: Basal, Luminal A and Luminal B.Example 5.4: BRCA Level 3 RNA-seq expression profiles downloaded from TCGA in January 2016 and consisting of 397 samples are classified into 4 subtypes: Luminal A, Luminal B, HER2 and Basal. This last test was not present in24 and it has been added in order to compare the classification performances also on RNAseq data.We tested Hope4Genes on five examples, chosen to explore the performances of the algorithm when the sample size, the number of classes and the profiling platform of the input data change. We thus considered multi-class and two-class classifications, data profiled with three different microarray platforms  and RNAseq, data composed of different sample sizes (from 12 to 397). The majority of the considered tests have been obtained from the original publication of Nearest Template Prediction (NTP)24.The used gene signature has been constructed inhttp://www.broadinstitute.org/cgi-bin/cancer/datasets.cgi. The size of the signatures ranges from 19 to 956. A detailed summary is reported in Supplementary Table NTP was also applied to the same examples and the performances of the two algorithms are compared in the following sections. The datasets, class labels and signatures used for the comparison were downloaded from www.broadintitute.org/genepattern)34.NTP was applied with standard parameters (cosine distance and 1000 permutations) through the GenePattern analysis toolkit , Weighted Voting (WV), Support Vector Machine (SVM) and k-Nearest Neighbor (k-NN). All the algorithms have been applied with standard parameters through the GenePattern analysis toolkit (www.broadintitute.org/genepattern)34. Differently from NTP and Hope4Genes these alternative approaches involve a training and test phases. Training datasets were downloaded from http://www.broadinstitute.org/cgi-bin/cancer/datasets.cgi. For Example 5.4 the Metabric dataset, available at https://www.synapse.org/#!Synapse:syn1688369 was employed as training set. For multi-class classifications (Examples 2–3 and 5) the expression levels were z-score transformed at gene-level to adjust the range of expression of each gene between training and test datasets.In line withFirst, we compared the classifications provided in the five tests by Hope4Genes against those of NTP without taking into account the FDR. As shown in Fig. We then compared the performances of the two algorithms considering only significant  associations. The FDR thresholds that we used are standard ones  plus the more stringent 0.5%. From this comparison we excluded examples 3 and 4 for which a perfect classification was already obtained without taking into account the FDR. The results obtained in the remaining 6 cases are summarized in the radar plots of Fig. More interesting are the results obtained in those two tests (examples 2 and 5.1) in which NTP had been found above to perform slightly better than Hope4Genes. Indeed, in example 2, while for high FDR thresholds (20% and 10%) NTP performs better than Hope4Genes, for an FDR of 5% Hope4Genes reaches the same classification performances of NTP and it finally outperforms NTP, for FDR values of 1% and 0.5%. In example 5.1, the use of the FDR completely overcomes the results obtained above. Indeed Hope4Genes results to perform better than NTP for all the five FDR thresholds, reaching a percentage of correctly classified samples greater than that obtained by NTP of 8–12%. These results confirm that the use of an Hopfield-like model and of its energy for the estimation of the FDR leads to an increase in classification specificity in respect to NTP.A comprehensive comparison of Hope4Genes and NTP (with and without FDR) together with more standard machine learning methods: CART, WV, SVM and k-NN is reported in Supplementary Table Finally, Hope4Genes also associates to each sample a secondary class giving a more exhaustive characterization of the expression patterns characteristic of each sample. Existing classification algorithms find a 5−10% of samples with no clear association to any class (i.e. unclassified). This frequently does not result from a limitation of the algorithm, but because at the molecular level these samples are a mixture of multiple classes. This holds true in all those contexts, as cancer subtyping, in which an evident distance among the classes does not exist. The secondary class provided by Hope4Genes thus helps to better characterize the molecular features of the samples under analysis.In recent years, the classification of cancer samples into subtypes has become one of the major issues of precision medicine. This is even stronger for cancer whose response to treatment regimes and disease course and severity vary going from one patient to another, even if both affected by the same tumor type. To achieve the best therapeutic performances, cancer patients need to be classified into subpopulations characterized by a common prognosis or response to treatment (“cancer subtype”). The association of a new patient to predefined subtypes, also known as class prediction, is thus vital to precision medicine. We here proposed Hope4Genes aimed at achieving this goal using a Hopfield-like model.We tested the performances of Hope4Genes in multiple test cases, exploring datasets profiled with different platforms, of different sample-sizes and multi-class and two-class classifications. In these tests Hope4Genes was found to obtain reliable results and to outperform the current state-of-art in the field (NTP), when the classification FDR is taken into account. These results prove that our implemented FDR computation based on the energy associated to the Hopfield model is able to better capture classification uncertainty than the NTP one based on signature genes resampling. Moreover, our algorithm also provides an extra information in respect to NTP, that is the secondary class, useful to characterize unclassified samples. Taken together the obtained results prove that the application of the Hopfield model for class prediction of gene expression data, interesting in itself, is also particularly well performing. The applications of Hope4Genes here proposed do not show the full potential of this model. Indeed the performances of the Hopfield model are maximized when dealing with signatures that are overlapping and redundant, as those required, for example, in the reconstruction of differentiation trajectories from single-cell RNA-seq data.35. Fast and precise class prediction algorithms, as the one here proposed, will be more and more fundamental to fully exploit the power of the available gene signatures. The use of class prediction algorithms will be required both in the context of genomic data analysis and in everyday clinical practice. Hope4Genes has thus the potential to impact not only computational biology, but also clinical decision-making, becoming a standard procedure applied by oncologists who need to choose the best performing therapy for a given patient.In the last years, the number of available gene expression signatures is getting larger36. Our approach could be thus naturally extended in the future for the integrative class prediction of multiple omics data. At the same time, given the results obtained by the Hopfield model in the context of single-cell pseudotime reconstruction, once stable signatures for the classification of single-cell RNA-seq data will be available, our algorithm could be also extended for the classification of these data.In this work, we only focused on single-sample class prediction for transcriptomic data, because mostly transcriptome is currently used to classify samples. However Hope4Genes can be in principle employed also to classify methylation and copy number data when signatures composed of epigenetic markers or chromosomes segments are availableN neurons connected through symmetric bidirectional links. We will denote the activity of neuron i with the binary variable xi ∈ {+1, −1}, corresponding to the two possible neurons states: firing (+1), or silent (−1). As a consequence, the state of the network can be represented as a binary vector x = (x1…xN) whose i-th component, xi represents the state of neuron i. The neurons interactions are encoded in the connection matrix, a N × N real symmetric matrix without self-interaction terms whose entries Wij define the weight of the connection between neuron i and j.The Hopfield model is a fully connected neural network able to recall stored memories starting from a noisy or distorted input. The Hopfield network consists of i at time t, the time evolution of the network is determined by the following updating rule:Denoting t, the state of all the neurons is updated simultaneously. Contrary, in asynchronous updating, at each time one node chosen at random changes its state depending on the state of its neighbours. We here chose to use asynchronous updating which has better convergence properties since it avoids spurious cycles.The dynamics described in  can be pp patterns μ = 1, …, p should be stable fixed points of the dynamics  and propose a slightly modified version of the Hebb’s rule (the net effect of this modification is to have a binary weight matrix with Wij = p/N if i and j belong to the same class and Wij = −p/N otherwise).1. To mitigate the effect of the non-overlapping organization of the stored patterns we extend the set of possible neurons states to Moreover we propose an asymmetric version of the updating rule. During the updating process, we let neurons switch from −1 to +1, but every time a neuron should be switched from +1 to −1 we instead eliminate it from the model (i.e. we set the corresponding neuron to zero). This is the reason why we call our version of the Hopfiled model asymmetric.The rationale behind this choice is that Hope4Genes suggests to switch a gene from +1 to −1 if the gene is bringing a misleading signal in the classification problem. This happens for example when a gene in input is over-expressed for purely stochastic reasons. We chose in this case to control the noise by not treating the gene as under-expressed (i.e. set it to −1), but simply neglecting it (i.e. set it to 0) from the model. Additionally, when a multi-class classification problem is considered, a sample may have no clear association to a specific class being a combination of different classes among which there is a dominant one. Our asymmetric version of the updating rule allows the algorithm to more efficently identify the dominant class by eliminating from the game genes which are correctly overexpressed in the input (i.e. they belong to the dominant signature and appear with sign +1 in the input) but are wrongly switched off due to the combined pressure of the genes of the other signatures. Setting them to zero we avoid the amplification of this wrong switch in the following updating steps. At the same time, due to the 95% overlap criterion discussed below, if too many of these events happen the algorithm is able to recognize that the input is too ambiguous and returns an “unclassified” result in output.Nμ of the signature associated to class μ is greater than Γ, Γ genes are selected at random from the corresponding signature. When Nμ > Γ all the neurons corresponding to genes deleted from the signature, are set to zero in the input configuration and, due to the updating rule eq. (M ≫ 1 times (default M = 100). For each sample, the class to which the Hopfield model converged in the majority of the M tests is selected as final output of our analysis.2. To overcome the presence of unbalanced signatures, which might induce a bias in favor of the larger signatures, we introduce a cutoff Γ on the signature size. Therefore, if the size rule eq. , they doThe choice of the threshold Γ is of crucial importance for the performances of the algorithm. On one side it should be chosen of the same order of the lowest signature size to balance the various classes. Moreover, small Γ implies small signatures that are generally less affected by noise (i.e. misclassified genes) than large ones.i is unstable isN = N(Γ). Given that a pattern is composed of N bits the probability of having an error-free recall of a stored pattern is (1 − Perror)N, that we require to be greater than some set value, such as 0.99. Keeping the binomial expansion and the lowest terms in Perror, given that this should be small, we get:On the other side Γ should be chosen as large as possible to maximize the number of error-free patterns to store into the model. Indeed, in a Hopfield model the probability that any bit p ≪ N and thus In our case, N satisfy equation p ≪ 6, according to equation N(Γ) turns out to be in between 70 and 100. Since the configurations we are storing in the network are correlated, we decided to select a value of Γ so as to have a value of N significantly larger than the above threshold and chose as default value for our analyses Γ = 200.Therefore Γ has to be small to avoid noise and to have balanced signatures, but at the same time it has to let https://github.com/lcan88/Hope4Genes, is composed of five main steps:Fixing the synaptic weights of the modelDataset preprocessingClassification of the samplesClassification confidence computationSelection of the secondary classHope4Genes represented in Fig. Fixing the synaptic weights of the modelp non-overlapping sets of Nμ, μ∈{1, … p} genes but in principle our algorithm could address also overlapping signatures. Starting from this first input, we associate to each class μ a pattern ξμ, corresponding to the class template, i.e. its components i is in the signature of class μ and The first input of Hope4Genes is the list of genes that are part of at least one signature. In all the examples that we studied the signatures were composed by g to eq. .The second input is the threshold Γ which may be chosen by the user. As discussed above the performances of the algorithm may depend on Γ. The suggested value that turned to be optimal in all the tested examples is Γ = 200.Dataset preprocessingWe provide an overview of the algorithmic steps below.A which is preprocessed, in accordance with4, as follows:All the genes that are not contained in any signature are filtered out.i, we associate to the corresponding node in the network the initial stateFor each remaining gene A[i] is the expression value of gene i in the current sample. If A[i] = 0  then we set 3.Classification of the samplesp stored memories. Since some of the neurons are set to zero during the evolution, in most of the cases it is not possible to find a perfect match of the evolving network state with one of the memories, therefore we assume that convergence is reached if 95% of the signature neurons match with a class template. The whole classification process is repeated M times with different random choices of the Γ genes for all the classes in which Nμ > Γ. Finally, each sample is associated to the class to which it converges in the majority of the M replicas. In the algorithm M is fixed to 100, but the results do not depend on the choice of M as far as M ≫ 1.We let the initial configuration mentclass2pt{minim4.Classification confidence computationIn order to asses the solidity of the predictions made by Hope4Genes we also compute the prediction confidence associated to each sample through the use of the False Discovery Rate (FDR).For each sample, the energy of the Hopfiled model (defined by eq. ) at the 5.Selection of the secondary classThe third input is the gene expression dataset One of the interesting features of our algorithm is that, when dealing with multi-class classification problems it allows to obtain the second best prediction. This prediction is obtained by deleting all the genes belonging to the signature of the convergence class and then repeating all the steps discussed above. The identification of the secondary class may be important in all those cases in which, due to the intrinsic characteristics of the analyzed sample, we expect a certain mixing of different classes as it happens for instance for tumor subtyping.Supplementary Information


In the era of precision oncology and publicly available datasets, the amount of information available for each patient case has dramatically increased. From clinical variables and PET-CT radiomics measures to DNA-variant and RNA expression profiles, such a wide variety of data presents a multitude of challenges. Large clinical datasets are subject to sparsely and/or inconsistently populated fields. Corresponding sequencing profiles can suffer from the problem of high-dimensionality, where making useful inferences can be difficult without correspondingly large numbers of instances. In this paper we report a novel deployment of machine learning techniques to handle data sparsity and high dimensionality, while evaluating potential biomarkers in the form of unsupervised transformations of RNA data. We apply preprocessing, MICE imputation, and sparse principal component analysis (SPCA) to improve the usability of more than 500 patient cases from the TCGA-HNSC dataset for enhancing future oncological decision support for Head and Neck Squamous Cell Carcinoma (HNSCC).Imputation was shown to improve prognostic ability of sparse clinical treatment variables. SPCA transformation of RNA expression variables reduced runtime for RNA-based models, though changes to classifier performance were not significant. Gene ontology enrichment analysis of gene sets associated with individual sparse principal components (SPCs) are also reported, showing that both high- and low-importance SPCs were associated with cell death pathways, though the high-importance gene sets were found to be associated with a wider variety of cancer-related biological processes.MICE imputation allowed us to impute missing values for clinically informative features, improving their overall importance for predicting two-year recurrence-free survival by incorporating variance from other clinical variables. Dimensionality reduction of RNA expression profiles via SPCA reduced both computation cost and model training/evaluation time without affecting classifier performance, allowing researchers to obtain experimental results much more quickly. SPCA simultaneously provided a convenient avenue for consideration of biological context via gene ontology enrichment analysis. Data generated for standard clinical oncology care has expanded exponentially. In addition to well-known clinical variables like symptoms, stage and histology, tumor specimens are now routinely sequenced for a range of mutations that may be more or less well characterized. These molecular profiles may suggest sensitivity to a range of molecularly targeted agents. Furthermore, high resolution, functional and molecular imaging methods (like CT-PET and MR) create quantitative metrics described through radiomics features. These also suggest profiles that can guide intervention and response. To facilitate the development of novel clinical decision support tools for oncologists, we have used the publicly available data characterizing head and neck squamous cell carcinoma (HNSCC). These profiles present a large data analysis problem necessitating the use of machine learning, dimensionality reduction, and biological pathway analysis techniques. We utilize machine learning classifiers to predict patient two-year recurrence-free survival and evaluate a variety of feature sets to discover potential useful clinical biomarkers. Feature sets include combinations of patient clinical and molecular data. To improve utility of this dataset for oncological decision support, imputation and dimensionality reduction methods are used to transform feature sets to more usable, interpretable forms.Current HNSCC literature often focuses on association of regulation of specific genes with prognosis , 2. OtheThe Cancer Genome Atlas (TCGA) Research Network  is a cooLarge, multi-institutional datasets present a variety of challenges to the development of methods and tools for clinical decision support. Namely, several clinical data fields in TCGA-HNSC offered issues of sparsity and inconsistency. Out of 15 identified clinical characteristics relevant to treatment regimen, none were populated for every patient. More specifically, the number of cases  with missing or unavailable data for these fields ranged from 88 to 504, with a mean of 349.5 and median of 342 cases lacking data for each field. In addition to the problem of missing data, several fields were populated inconsistently, with responses varying both due to human error (e.g. leading zeros in numeric fields) and varying convention . Such complications required extensive preprocessing and an expert system built using domain-specific knowledge to determine whether each patient had received a specific type of therapy. Even after this preprocessing and condensing of treatment fields, issues of missing data persisted. Whether a patient had received radiotherapy and/or chemotherapy was unclear for 47 and 27% of cases, respectively. One possible technique for handling such problems is to exclude cases or variables with missing data, as was done previously with this dataset . Due to Molecular datatypes are often extremely high-dimensional. Feature selection and dimensionality reduction techniques are necessary steps when utilizing such data to best employ available computational resources. There are several strategies for selection and dimensionality reduction, including feature filtering, feature transformations, and wrapper methods such as sequential selection . In thisPerforming imputation on large datasets requires development of an expert model. Careful examination and correction of inconsistent and missing values was performed in collaboration with expert oncologist Dr. John Buatti . A rubric for consistent preprocessing and condensing of the 15 relevant treatment fields resulted in a much more concise and usable dataset. However, a significant fraction of the TCGA-HNSC patients still had uncertain status in their treatment regimens. To address this, Multivariate Imputation by Chained Equations (MICE) [MICE builds predictive models for each missing variable to realistically impute entries based on the remaining predictors. It takes into account uncertainty during imputation, allowing it to outperform single imputation methods , 17. ForNaïve Bayes and Random Forest (RF) classifiers were selected for this evaluation. The Bayesian model provides a pure conditional-statistical effort to predict survival, though it does not consider interaction effects. The Random Forest model, being a set of recursive partitionings, extensively leverages interaction effects. Earlier work suggests RF models are effective for this classification problem . The preIn the TCGA-HNSC dataset, solid-tumor expression was available for 520 of the 528 patients. With a feature set of 20,531 solid-tumor RNA expression variables, seven tumor grading variables, and the random forest-imputed treatment variables, several RF classifiers were trained to predict two-year recurrence-free survival. The classifiers varied in feature sampling and tree construction procedures: a standard RF, a weighted subspace RF (WSRF) , and a cWith the full set of RNA expression data, the feature set was first refined through two filters: a univariate, near-zero variance filter to remove uninformative features and a multivariate correlation filter to remove features with correlation greater than 0.9. These filters removed several thousand expression variables from the feature set.In addition, a dimensionality reduction was performed via Sparse Principal Component Analysis (SPCA)  which haAfter training, variable importance for the 10-component SPCA feature set was evaluated for each of the classifiers. The genes that comprise the most and least important variables were examined with a gene ontology enrichment analysis (GOEA) , 23. AnaA nested cross validation (CV) is used to tune hyperparameters and estimate out-of-sample performance, a measurement of how well a classifier would generalize if it were to be trained on the entire dataset. Data is split into ten folds (or partitions) for the outer CV. In each iteration, one fold acts as the testing data and the remaining nine folds act as the training data. Within each fold of the outer CV, a repeated grid search CV procedure  is carriFor the missing data imputation, models were trained and tested in Weka 3.9.1  with 10-In the RNA expression experiments, three different RF classification procedures are considered. Classifier training and evaluation was handled using the R package caret . ClassifTo evaluate classifier out-of-sample performance, the area under the receiver operating characteristic curve (commonly denoted AUC) is employed. AUC is a very popular and commonly used classifier metric in the literature with an intuitive probabilistic interpretation: AUC is the probability that the classifier will score positive observations higher than negative observations. Mathematically, an AUC of 0.5 is equivalent to random guesses and is the standard baseline for this metric.Feature importance is a measurement of how perturbations to variables affect classifier performance. A conditional variable importance procedure has been applied in this work. Conditional importance involves not only univariate perturbations, but conditional perturbation of variables and the variables with which they correlate . For theAs shown in Table The relative importance of treatment features doubled as a result of imputation  allow comparable classifier performance while drastically reducing runtime and necessary computation. Though not quantified here, memory requirements were also much lower for the dimensionality-reduced data. Additionally, this reduction allowed us to identify gene set candidates for GOEA. In both important and not-important SPCs, the GO term “cornification” is found, indicating that this biological process is related to high-variance genes in this dataset. Terms found only in the high-importance SPC gene sets are related to cell motility , immune response, cell growth, and blood vessel development. Activity of genes involved in these processes could be indicative of a cancer’s ability to survive, grow, and metastasize, suggesting that these SPCA transformed RNA data contain useful information about underlying relationships between solid- tumor expression and two-year recurrence-free survival.In modern oncological research, TCGA datasets present significant large data analysis challenges, from clinical parameter sparsity to high dimensionality. Facing these problems requires significant preprocessing and machine learning modeling to uncover new knowledge. A multivariate imputation method (MICE), SPCA dimensionality reduction, and an SPC-focused GOEA are presented in the context of TCGA-HNSC clinical and RNA expression variables to improve usability of data for future HNSCC decision support.vAs others , 31 haveUnsupervised transformation of RNA expression data via SPCA was extremely useful in improving interpretability of survival models and biomarker identification, by limiting the number of genes contributing to each principal component and allowing for a more nuanced examination of the underlying biological processes. The biological processes found to be associated with only high-importance SPCs may be useful in future feature vselection for biomarker discovery. Additionally, the SPCA functioned well as a dimensionality reduction technique, as the dimensionality reduced features allowed for significantly lower computation time without significantly affecting classifier performance. From the literature and these analyses, unsupervised transformations of RNA expression data seem a viable option for future integration of molecular data into HNSCC clinical predictive models.Future work will consider the effect of clinical imputation on models also utilizing molecular data, both with SPCA transformations and other unsupervised feature transformations methods such as denoising autoencoders. Additionally, biomarker evaluation will be expanded to directly consider right-censored survival.


It proved superior to the single-gene and alternative enrichment analyses in terms of (1) universal applicability to different data types with a possibility of cross-platform integration, (2) consistency of the discovered correlates between independent drug screens, and (3) ability to explain differential survival of treated patients. Our new screen of anti-cancer compounds validated the performance of multivariate models of drug sensitivity. The previously proposed methods of enrichment analysis could achieve comparable levels of performance in certain tests. However, only our method could discover predictors of both in vitro response and patient survival given administration of the same drug.Despite the widening range of high-throughput platforms and exponential growth of generated data volume, the validation of biomarkers discovered from large-scale data remains a challenging field. In order to tackle cancer heterogeneity and comply with the data dimensionality, a number of network and pathway approaches were invented but rarely systematically applied to this task. We propose a new method, called NEAmarker, for finding sensitive and robust biomarkers at the pathway level. scores from network enrichment analysis transform the original space of altered genes into a lower-dimensional space of pathways. These dimensions are then correlated with phenotype variables. The method was first tested using  This was aggravated further when novel omics platforms expanded the variable space from thousands to nearly millions of potentially informative molecular features. In addition, profiling of cancer samples revealed that genomic alterations across tumors of the same type appear disparate and poorly overlapping2. As a result, variability between cancer samples is often higher than is assumed by the common parametric statistics3. Beyond a few success cases5 molecular cancer signatures have been hard to corroborate in a novel, independent cohort. Across a number of meta-analyses, conclusions about practical applicability of the signatures range from entirely negative7 to mixed or moderately positive8. The common understanding is that seemingly disparate individual events must be confluent to certain pathways that represent cancer hallmarks and pathways9.The problem known as the “dimensionality curse”in vitro was questioned by finding that molecular landscapes of cancer cell lines are be very different from those of original tumors10. A later, more comprehensive exploratory analysis demonstrated overall consistence of molecular aberrations between cell lines and primary tumors from matching cancer sites9 – although these authors did not investigate the therapeutic relevance of discovered in vitro correlates. Haibe-Kains and co-authors published a discouraging comparison11 between two large in vitro screens13. After that conclusion and the following polemics14 and failures to replicate particular findings15, the urgent need in cross-platform and clinical validation became even more apparent. It is dictated by both statistical and biological challenges, such as excessive data dimensionality, imperfect analytical tools, the heterogeneity of cancer genomes, and the downstream diversity of methylation and expression patterns16. Authors of one of the most up-to-date investigations still admitted that the ability of cancer cell drug screens “to inform development of new patient-matched therapies… remains to be proven”17. On the clinical side, oncologists expected reports on patient-specific alterations in the light of knowledge available from computerized support systems18. In our view, these challenges could be most systematically addressed by summarizing sparse, disparate events at the pathway level via the global interaction network.Modeling drug response 19. One particularly winning strategy was to employ multigenic expression patterns. Such ‘meta-genes’20 were, despite the seemingly ‘network-free’ definition, nothing other than modules in a co-expression network, which allowed dimensionality reduction and a biological generalization. Another DREAM project revealed efficiency of summarizing gene expression in cancer cell lines over pathways21.Adding omics data to clinical variables has demonstrated the potential for prediction of cancer disease outcome in a DREAM challengeFurther, identifying patient sub-categories responsive to a treatment is more challenging than one-dimensional drug sensitivity or survival analyses. A practical method should profile individuals across the cohort, so that the profiles can be fit to clinical variables and covariates. Therefore, a crucial feature for biomarker discovery would be the ability to assign scores to individual samples rather than to derive feature-pathway associations from the whole data collection. In addition, further sample classification in a flow of new patients should not require re-running the analysis on the whole cohort, i.e. recalculating the data space, as is often the case.22. Using that approach of significance estimation via comparing network connectivity to a null model, NEA24 can characterize experimental and clinical samples with pathway scores by accounting for sample-specific gene set relationships in the global gene interaction network. The pathway-level output could be used in downstream analyses against arbitrary phenotype models. The ability to summarize rare alterations that cause the recurrent cancer phenotypes into pathway profiles provides higher statistical power, more information on the underlying biology, and robustness in phenotype prediction. However, neither NEA nor alternative methods of pathway enrichment had been systematically applied to the task presented above: the discovery of biomarkers suitable for individual outcome prediction.In this work, we use acronym NEA to refer to a specific approach for network enrichment analysis, which ascends to the idea of accounting for the node degrees of individual genes26, signaling pathway impact analysis (SPIA27) and EGSEA28 that integrates multiple enrichment analysis methods. The outline  as well as in eight TCGA clinical cancer cohorts, (2) investigate preservation of this content across drug screens, (3) perform a novel, small scale drug screen and demonstrate that the pathway-level multivariate models withstand the independent validation, and finally 4) validate the identified correlations in clinical treatment profiles from TCGA30 , we discuss their principles, consider both applicability to biomarker discovery and software usability, and motivate our choice of methods presented in Fig. ine Fig.  and deta31  as well as two options of EGSEA, a method that combined multiple existing algorithms. In sections 3…5 of Results, we report the outcome of systematic analyses of the experimental datasets under these alternatives  and then retaining only those with significant network enrichment against the rest of the set. This approach36 had been proposed for distinguishing between driver and passenger mutations, so that the filtering should enrich AGSs in cancer driver genes. Next, AGSs from gene copy number and expression data included genes most deviating from the cohort means. This was achieved by using one of the three alternative algorithms (see Methods). Again, even such deviant gene sets could still be too large, e.g. due to listing copy number-alterations over extended chromosomal regions. In order to compact these, alternative AGS versions were derived by retaining only genes with significant network enrichment for signaling and cancer pathways or for the mutation-based AGS of the same sample, which reduced the AGS lists 3–10 fold. Finally and as an extra option, we tested combined AGSs, produced by concatenation of the platform-specific AGSs.In order to analyze data from the 37 and cell lines’ sensitivity to each of the 203 anti-cancer drugs from Basu et al.29. The fraction of low p-values (e.g. p(H0) < 0.001) in the total number of statistical tests did not exceed the level expected under “true null”, i.e. in absence of any associations. Therefore, no genes received q-values 38 below 0.05. On the contrary, the correlation analysis of gene expression12 against the same drug sensitivity profiles discovered nearly 15,000 patterns of association between gene expression and drug sensitivity  with p(H0) < 0.001. After the adjustment, more than 2500 of these gene-drug pairs remained significant at q < 0.001. These two examples demonstrate how dramatically the information content could vary depending on the feature type and data origin.The goal of this first, exploratory analysis was to compare the different methods and feature classes in their ability to explain the differential drug sensitivity. To this end, we counted how many features were significantly associated with a phenotype after adjusting for multiple testing. For example . Each case was tested on both relapse-free and overall survival and three follow-up intervals. We also analyzed the relative performance of different AGS classes  and individual gene node level (GNEA) contained either approximately the same or larger amounts of information on drug sensitivity compared to the original gene profiles Fig. . We alsoWhile the original features manifested considerable correlations in a number of classes, fractions of significant correlations were largely inferior when compared to NEA classes. Preliminarily, the different methods could be ranked by potential sensitivity in the following order: original gene profiles < [either ORA or EGSEA] < [either ssGSEA or SPIA] < [either PWNEA or GNEA]. However, we did not draw ultimate conclusions from the ranking. This exploratory analysis only informed us on the Type II error rates . It suggested that multiple alternative methods and data types could appear – by following the state-of-the-art approaches – potentially predictive of drug sensitivity. In order to evaluate robustness of these predictions we proceeded to the validation step as described below. We emphasize that only the AGS-based methods ORA, PWNEA, and GNEA enabled using discrete (gene point mutation and copy number) data and allowed integration over different platform types – while the other alternatives could be applied only to expression platforms.in vitro drug screens: CCLE12, CGP13, and CTD29. Similarly to an earlier presented comparison between CCLE and CGP screens11, we found that the association values between drug sensitivity and original features only weakly agreed between the drug screens.In order to test reproducibility of the drug-feature associations in alternative experimental settings, we used data from three 13 and CTD29 screens, the correlation values between Affymetrix expression data and sensitivity to navitoclax ranged from R = 0.31  to R = 0.81 (GNEA). More comprehensive analyses demonstrated . Activity of these compounds was re-tested in a panel of 20 cancer cell lines (the ACT set) for which gene expression and point mutation profiles data were available from the CCLE. Similarly to the results in Fig. Next, we validated drug sensitivity profiles of three anti-cancer compounds, tested previously in the CTD screen - RITA, PRIMA-139. Starting from all available features, each model was finally reduced to a much smaller subset. Multivariate models are notoriously prone to over-fitting when the number of variables exceeds the number of samples. For this reason, validation on independent sets has become an essential requirement in such studies40. The CTD-based models were thus created using cell lines not found in the ACT screen. The comparison was also streamlined by using only the data from CCLE Affymetrix and point mutation datasets versus two respective feature AGS classes mutations.mgs and significant.affymetrix_ccle. Using other classes produced similar results (data not shown).As shown above, the original gene profiles were poorly preserved across drug screens. Therefore, we compared the CTD results with those from ACT screen in a more relevant multivariate approach using the “elastic net” methodFigure Overall, the performance of the original profile models on the validation sets appeared comparable to that of PWNEA. However importantly, the former had much more freedom in model term selection since the initial feature space was around two orders of magnitude larger than that in PWNEA. Consequently, despite the rigorous cross-validation and feature selection implemented in the glmnet algorithm, using the original profiles generated more complex models  which fit the training sets better. At the validation step however, the performance of the original data models significantly worsened - whereas the PWNEA-based models performed at the same level . This result essentially corroborated the previous conclusion about higher robustness conferred by NEA, compared to the usage of original gene profiles.in vitro drug screens and clinical application of the same drugs. Any trustworthy setup of such an analysis would be very complex, so that even cross-validation and adjustment for multiple testing could not guarantee an unbiased probabilistic estimation. Thus, the final judgment should be made after a biologically independent ad hoc validation from the in vitro to the clinical domain. Even though the TCGA collection did not provide correctly balanced, randomized cohorts for estimation of relative risks, error rates etc., our task was simplified by only needing to compare the methods’ performance. In the eight largest TCGA cohorts, we counted how many significant in vitro-detected features correlated with survival of patients who received same drug30,  correlation with drug response in vitro tended to correlate in the same manner with the patients’ response. Due to the testing of alternative AGS classes, respective numbers of matching pairs in ORA, PWNEA, or GNEA were an order of magnitude higher than in raw data . Then we determined whether correlation p-values of individual features, in their turn, correlated between in vitro screen). Remarkably, the separate test using the 197 KGML KEGG pathways also demonstrated superiority of PWNEA over ORA, ZGSEA, EGSEA, ssGSEA, and SPIA - despite the reasonably good performance of the latter two on the in vitro datasets presented above. Thus at this crucial validation stage while translating drug sensitivity correlates between in vitro and clinical applications, robustness of the features increased in the following order: [point mutations and gene copy number changes] < [gene expression] <  < PWNEA < GNEA.At the next step . They were identified for docetaxel, gemcitabine, and paclitaxel in BRCA  = 4.2 × 10−11) and with overall survival of OV patients  = 4.4 × 10−4 during the 3-year follow-up time while accounting for “clinical stage” as a covariate).Carboxylesterases (CESs) are capable of hydrolyzing gemcitabineES1 Fig. , CES2, Cres Fig.  for EBAG46 and cytoprotection48. In our analysis its GNEA feature was  = 1.1 × 10−07) and with relapse-free survival of OV patients (p(H0) = 0.003).The intestinal-type alkaline phosphatase ALPI is known to modulate cancer cell differentiationwas Fig. , in paraThis setup could not eliminate possible confounding effects from multi-drug treatment history and clinical factors that might determine administration of specific drugs. Nonetheless, the NEA scores apparently explained the differential sensitivity to anti-cancer drugs in a much more robust and efficient manner than the original data.49. However we note that additional responders could be detected by using multiple markers in parallel. As an example, beyond the “NOD–like receptor signaling pathway” at Fig. A visual inspection of the survival curves in Fig. 51. The poor performance of the individual gene analysis and alternative enrichment methods could be explained by the excessive dimensionality of the former and poorer sensitivity of the latter. In addition, the ability to use smaller and hence more specific AGS could have provided extra advantage of NEA over ORA and GSEA. On the other hand, NEA could also deteriorate on AGS of insufficient size when using sparser networks (around 104…105 edges) and networks with many missing nodes. These potential limitations were established earlier36 and we tried to avoid them in the present work by using e.g. a denser network from data integration. We admit that a future, more comprehensive version of NEA might adopt advantages of the alternative enrichment methods by employing full gene lists (as in GSEA) and intra-pathway topology (as in SPIA). Indeed, at certain steps of our analysis these methods demonstrated performance comparable to that of NEA.In our view, the advantages of our NEA approach are due to the following features of network-based data interpretation: (1) combining major types of molecular interactions in a biologically relevant way, (2) summarizing seemingly disparate molecular alterations at the level of pathways and processes, and (3) enabling lower-dimensional statistical analysis. In addition, a network context, with different types of evidence behind the edges provides better grounds for biological interpretationin vitro versus clinical results, these classes were almost equally represented (Suppl. Table 2). We have also seen differences between different filtering approaches in AGSs of classes significant.mini and significant.maxi  – although the enzymes here work in an opposite direction: they activate irinotecan rather than degrade as they do to gemcitabine. Nonetheless, relevant GNEA scores might have been informative for response to irinotecan. The patients’ response was sufficiently differential, too: while all the irinotecan-treated patients relapsed, the time to relapse varied from 78 to 1265 days. However, we did not observe almost any sensible correlation of the pathway genes neither as GNEA features nor as raw gene expression profiles. In regard of GNEA, this elucidated a lack of network linkage between the AGSs of responders (or non-responders) to the irinotecan pathway.The individual molecular phenotypes of cell lines and tumors were characterized with AGSs compiled using a number of alternative methods. The analysis provided a primary comparison of their relative performance but – at the current stage – did not enable definite conclusions about performance of the different AGS classes. Indeed, AGS of fixed size (top.N) versus variable size (significant) compared differently in the cell lines versus the TCGA data (Suppl. Table 1). Further in the analysis of consistency ine Fig. , namely de novo pathways - similarly to the approach by Glaab et al.53, but specifically informative of the drug response or disease prognosis. An example of such a functional set could be the presented above combination of the ten carboxylesterases and cytochromes.Further, our FGSs were created by third-party sources and never meant to be used in NEA. Thus, another step for NEA-based biomarker discovery would be the compilation of novel, specifically optimized FGSs. Ultimately, one could compile Finally, given the low overlap of member genes between individual AGS, it is important to establish how AGS-level biomarker panels would practically summarize gene-level information and organize the accompanying statistical framework. Ways to compile and employ multi-platform AGSs, optimal FGS design, and construction of NEA-based biomarker panels should therefore become the topics of future studies.in vitro phenotypes in corresponding clinical applications was possible by using our method but not by original profiles or alternative methods.We have presented method NEAmarker for using network enrichment scores in prediction of drug response and demonstrated its advantage compared to the conventional analyses of original gene profiles and alternative, previously presented enrichment methods. In the first place, NEAmarker allowed combining data from multiple omics platforms. Further, the NEA scores indicated higher statistical power to detect enrichment and were therefore more prone to manifest correlation with drug sensitivity. The higher robustness, anchored in the network context, enabled better preservation better between independent screens. Multivariate models using NEA scores were built from a lower-dimensional space, thereby proving more compact and, at the same time, robust when re-tested on novel data. Finally, corroborating In this analysis at ACT (“Advanced Cancer Therapies”) centre, we used 20 cancer cell lines for which molecular data could be found in the CCLE Affymetrix set as well as in both CCLE and COSMIC point mutation sets: A375, HCT116, HDLM2, HT29, JVM2, K562, L428, MCF7, MDAMB231, MV411, NB4, PL21, RAJI, RKO, SJSA1, SKBR3, SKNAS, SW480, T47D, and U2OS. Eight of these cell lines had also been included in the CTD screen . In order to avoid overlap in the multivariate models, we excluded these eight cell lines while training the original models from the CTD data and only used them in the validation set.Cell proliferation was estimated with the WST-1 assay (water soluble tetrazolium). Briefly, cells were incubated with each drug for 72 hours in a 96-well plate. At the end of this period, they were incubated with WST-1 reagent (Roche) for 2 hours. Absorbance at 450 nm was measured following the instructions from the manufacturer. The cell proliferation rate compared to that in the control was calculated.log3(10) ≈ 2.09. The cell lines MCF7, RAJI, SJSA1, SKBR3 did not show any sensitivity while using the initial stock (IC50 = 0), so that their IC50 values upon JQ1 treatment were declared missing.For adherent cultures, cells were attached overnight before adding the compounds. For hematological malignancies, the compounds were added simultaneously with seeding cells. The initial cell density was chosen so as to avoid confluence at the end of the assay. Each compound was applied in six consecutive 3-fold dilutions. In RITA and Apr-246/PRIMA-1-met, the stocks were established at the concentration based on individually determined efficacy. Final concentrations were for RITA: 0.01, 0.04, 0.12, 0.37, 1.11, 3.33 µM and for Apr-246/PRIMA-1-met: 0.3, 1, 3, 9, 28, 83 µM. For JQ1 the cell lines HDLM2, HT29, MCF7, RAJI, RKO, SJSA1, SKBR3, and SW480 were tested using the final concentration range 1.66…0.007 µM in 1:3 serial dilutions. However later we found it necessary to raise the concentration by one order of magnitude, so that the final concentrations for the rest of the cell lines were 16.66, 5.55, 1.85, 0.61, 0.20 µM. Then we respectively adjusted IC50 values for the first group as if they were tested under the final concentrations. This was done by incrementing the initial-stock IC50 values of HDLM2, HT29, RKO, and SW480 by IC50 was defined as the drug concentration inducing a 50% reduction in cell proliferation compared to the control. In the quantitative analysis, we used a universal scale for all the drugs where units 1…6 stood for dilution steps . Sensitivity to compounds was expressed in IC50 values varying from 0 (insensitive to compound) to 6 (fully sensitive to compound).48. The model form (argument fct) was chosen as LL.4, where model parameters Lowest and Highest were fixed at cell proliferation rates 0% and 100%, respectively, while parameters slope and IC50 were left unfixed. The IC50 values are provided as Supplementary File IC50values.ACTscreen.xlsx. Missing values are presented as NA.IC50 values and p-values of the model parameters were calculated using function drm from R package drcet al.12 analyzed cell line sensitivity to 24 drugs in 504 cell lines. These authors considered a range of numeric sensitivity metrics for their analysis and finally preferred ‘normalized activity areas’. These original units were calculated as areas under compound response curves where higher values corresponded to higher sensitivity so that 0 stood for ‘insensitive to compound’ and 8 corresponded to ‘full sensitivity’. Further, the activity area values were normalized for unequal luminescence in the assay. We rendered them normally distributed by log-transformation. Thus the values in our analysis range from −3.00 meaning ‘insensitive to compound’ to +2.31 meaning ‘maximal sensitivity’.Barretina et al.13 analyzed 138 drugs in 714 cell lines. They used a combination of IC50 and the slope parameter to achieve the most complete description of responses. We decided to use the AUC as a single feature that reflects the both values. AUC was originally provided in the same table and ranged from 0% (fully sensitive) to 100% (insensitive). To approach the normal distribution, we transformed the values as log(1 - AUC), so that now they ranged from −8.11 meaning ‘insensitive to compound’ to 0 meaning ‘maximal sensitivity’.Garnett et al.)29 mainly used areas under curve (AUC) for their quantitative analysis of 203 drugs in 242 cell lines. We reproduced this approach in our study. In completely insensitive cases, the full area under eight experimental points reached 8, whereas 0 stood for full sensitivity. Thus, the scale of this screen was inverted compared to the other screens, which was considered in all calculations.The authors  and cell lines (622 vs. 1034). For these reasons, we used the CCLE dataset in all the presented alternative analyses except NEA and ORA. AGS for the latter two were compiled from the both platforms (affymetrix_ccle and affymetrix_cgp). Expression values x of the downloaded datasets were transformed to log2(x).The profiling was performed in CCLE study using Affymetrix GeneChip® Human Genome U133 Plus 2.0 Array and in CGP study by Affymetrix GeneChip® HT Human Genome U133 Array plate. The expression datasets were normalized as described when made public by the authors. Expression profiles in the CTD study were from CCLE. It has been shown earlier12 for 994 cell lines. In addition, we downloaded COSMIC data37 independently produced by the same platform and then post-processed in three different ways to provide total, absolute copy number per gene, number of copies of the minor allele, and a binary classification of gene copy number values into “gain” vs. “loss”. All datasets were used as downloaded, without further processing or normalization.CCLE, CGP, and CTD all employed Affymetrix SNP 6.0 microarrays for gene copy number detection. We downloaded the CCLE datasetCCLE provided point mutation data on sequencing of 1667 genes in 904 cell lines. In addition, we downloaded COSMIC data from exome sequencing of 1023 cell line genomes, which mapped to 19759 gene symbols. Mutation data from the both screens were used in the binary form, i.e. all specifying attributes were neglected.Following the same approach, we employed TCGA data on somatic point mutations reported in MAF files. The column ‘Variant_Classification’ contained a number (more than 15) different codes, most frequent being Missense_Mutation, Nonsense_Mutation, and Silent. the latter constituted around 25% of the total number of somatic mutations reported in the eight cancers, while around half of such cases contained only mutations reported as silent. This fraction would not significantly affect the false positive and true discovery rates in any enrichment analysis. Furthermore, we found that in each cohort tens to hundreds of most frequently mutated genes (e.g. top 5% ranked by frequency per base pair length) had a significant rate of purely silent mutations (see Supplementary File Nmut_vs_frequency.4cohorts.pdf). Discarding such would exclude many potential cancer drivers – although of yet unknown mechanisms. We therefore included all mutation records present in the MAF files.We evaluated a number of existing multivariate, enrichment-based, and/or network analysis methods that could be potentially useful in the proposed analysis, accounting for their complexity, applicability to different experimental designs, and the ability to analyze individual samples rather than the whole cohort. Various statistical algorithms have been proposed to quantify functional relevance of pathways and other gene sets by accounting for gene network topology.54, that edge confidence weights should influence penalties on the model coefficients55, or that there exists equivalence  between connectivity of nodes and covariance of model terms57. Advanced regularization of linear models in these methods often demonstrated promising efficiency58. However being very sophisticated, these models proved hard to tailor to novel, specific experimental designs. Notably, it was not feasible to include additional covariates or interaction terms which would be necessary for e.g. analyses similar to the one described in the present work - not even in the dedicated survival analysis method DegreeCox59. Using pathway membership information for summarizing cross-pathway linkage was proposed in60 - however, adjusting its error rate model to other purposes has not been straightforward.A number of methods can generate sparse regression models via network-based regularization, i.e. account for topological relations between potential predictors . The regularization is based on certain assumptions, such as that e.g. term coefficients of neighbor nodes should be zeroes or non-zeroes simultaneously64, also called “class I” in the classification by Huang et al.31. For comparison, the most popular gene set enrichment analysis, GSEA25 has been usually applied to finding pathway enrichment in gene lists pre-ranked by cohort-wise statistics. As an example, Haibe-Kains et al.11 analyzed correlations between drug sensitivity and molecular features calculated on whole in vitro drug screens13 which are among the datasets re-analyzed in this article. Those pathway enrichment scores represented correlates of drug sensitivity over the whole screened collections rather than characterized individual cell lines. Likewise, Iuliano and co-authors65 matched molecular landscapes to survival in cancer sample cohorts in order to reverse-engineer relevant pathway and network structures. Thus, global methods often employ powerful, heavily optimized statistical techniques and are used for sample exploration or differential expression analysis66 but cannot serve features for phenotype prediction in novel cell lines or tumors. An overview of network applications in cancer studies52 showed that, indeed, most of the existing methods enabled exploratory analyses, discovery of driver genes and pathways as well as splitting a cohort into molecular subtypes, but did not characterize individual cases.Technically, individual scores that estimate samples’ uniqueness as compared to the rest of the collection can be obtained already from ORA, i.e. from the simplest analysis of dichotomous 2 × 2 tables applied to sample-specific gene sets27 and iPAS67 were also capable of calculating sample-specific pathway scores. However, their scores were based on gene expression values, which excluded the using of other data types. A genuinely integrative multi-omics method PARADIGM68 , on the contrary, accounted for combinations of events in the chain DNA->mRNA->protein activity. As input, it required well characterized regulatory relationships – a complete set of which would rarely be available. Also, similarly to the former group of methods, it relied on comparing cancer to normal samples. Those dramatic alterations between the normal and cancer tissues encompassing thousands of genes would mask more fine-grained features that determine between-tumor heterogeneity, differences between sensitive and refractory cases etc. This requirement also precluded analyzing data where normal matches are missing, such as the widely used in our analysis cancer cell lines. Finally, EnrichNet53 has been an algorithm closest in spirit to NEA: by using random walk with restart (hence not limited to 1-step network distances), it can trace AGS-FGS relationships via network paths. However it existed only in a single-AGS, web-based implementation and therefore was also not available for testing it the present analysis.A number of hybrid approaches, such as SPIAet al.69 have explored cancer cohorts with pathway-level sample scores derived from gene expression data in a quantitative way and found that certain sample clusters can be associated with patient survival. On the other hand, the network-based methods have been developed only recently and are therefore ‘too young’ to have been exploited fully. Above, we have also mentioned the network-based regularization of multiple regression models where inclusion of gene terms into the models was essentially coupled to their co-expression.Even though individual enrichment scores can be correlated with phenotypes, they have still been rarely used in predictor models. In the case of ORA and GSEA, the major reason was that the enrichment is mostly detectable for large FGSs (hundreds to thousands genes), but such are unlikely to characterize functional differences between tumors - while compact, specific, and discriminatory gene sets tend to escape their limits of statistical power. Nonetheless, Drier Using original gene profiles from respective omics platforms;ORA, over-representation analysis which was capable of working on exactly the same AGS and FGS as PWNEA;ssGSEA, ranking by absolute gene expression value,ZGSEA, ranking by deviation of gene expression from the cohort mean;GSEA on full ranked gene lists, applying two alternative methods:SPIA, measuring the pathway perturbation via known intra-pathway topology.EGSEA, that combined existing enrichment methods, using five out of the total twelve ones, which were capable of producing individual sample scores.We finally decided to include in our testing, in parallel with PWNEA (pathway level NEA) and GNEA (gene node level NEA), the following methods:Using GSEA and SPIA was restricted to only transcriptomics data. SPIA, in addition, could only be run on pathways with known topology, which limited the set of available FGS to 197 KEGG pathways available in KGML format. This created an additional, specific line of testing on a limited collection of input data and FGSs for the methods ORA, ssGSEA, ZGSEA, SPIA, and PWNEA see Figs .50 with consecutive merging of five more resources as described and benchmarked previously36. The results of that benchmark indicated that FunCoup was superior to STRING , mostly due to the latter broadly using prokaryotic evidence and therefore less specific in cancer-related analyses. The second conclusion from the benchmark was that adding to the FunCoup network edges of curated databases significantly improved its performance. We therefore added the FunCoup-based network with functional links from KEGG71, CORUM72, and PhosphoSite73, MSigDB transcription factor-related part74), and an own reverse-engineered network36. The resulting network thus combined a wide range of molecular mechanisms, functional relations, and metrics from high-throughput data sets: physical protein-protein interactions, membership in same protein complex, membership in the same pathway, correlation of mRNA profiles, correlation of protein abundance values, protein phosphorylation, coherence of GO annotations, concordance of upstream regulators (transcription factors and miRNAs), co-localization in same sub-cellular compartments, similarity of phylogenetic profiles etc. It contained 974,427 edges (links) between 19027 nodes (distinct human gene symbols).The network was based on the FunCoup methodet al.36).mutations.mgs: point-mutated genes that proved to be significantly NEA-enriched to either KEGG pathway set #05200 “Pathways in cancer” or to the full set of point-mutated genes annotated in the given genome . These AGSs had variable sizes, depending on the significance criterion.significant: most deviating from the gene’s cohort mean (same as above), but selected only if below the formal significance threshold .significant.filtered.maxi: members of the respective significant set were required to be significantly NEA-enriched to any of the signaling pathways  or to mutations.mgs set of the same sample.Point mutation data (mutation gene sets):significant.filtered.combined.mini: a merge (concatenation) of all sets of type significant.filtered.mini.significant.filtered.combined.maxi: a merge (concatenation) of all sets of type significant.filtered.maxi.Combined (multi-platform) AGS:For convenience, AGS labels refer also to the platforms and sources, e.g. top.200.cn_ccle, significant.filtered.maxi.affymetrix_ccle etc.The functional gene sets, FGSs, were AGS counterparts in the analysis. The main collection of 328 FGS was based on the KEGG pathways, the full collection of which was complemented with a number of separately published cancer pathways as well as specific GO terms corresponding to cancer-relevant signaling or hallmarks of cancer . Another approach was applied to enable compatibility with GSEA and SPIA. These methods were designed and are most suitable for analyzing expression data and, apart from that, SPIA was applicable only to pathways with well characterized intra-pathway topology. We therefore employed a special set of 197 KEGG pathways for which the topology was available in KGML files and tested on it SPIA, GSEA, ORA, and PWNEA exclusively gene expression data . The analysis on the FGS collection is referred to as pathway-level NEA (PWNEA).In the other version of our analysis, called gene-wise NEA (GNEA), we treated each of the 19027 network nodes, regardless of their pathway or GO annotation, as a single-gene FGS.23. In the current implementation, we evaluated enrichment of AGS versus FGS by the formula:n means “complement to n”, i.e. all global network edges that did not belong to NAGS-FGS. The number of links expected under true null, i.e. by chance, was determined by:The major principles of NEA were described earlierNAGS and NFGS reported the sums of connectivities of member nodes of AGS and FGS, respectively, and Ntotal was the number of edges in the whole network. Since it was desirable to provide normally distributed values for the downstream analyses , we calculated p-values from the Χ2 statistic p(H0) = f(Χ2) using function pchisq available in R language and then re-calculated corresponding z-scores from the p-values as Z = F(p(H0)) with function qnorm. Since Χ2 is only defined on the non-negative domain, the z-scores were coerced negative in cases of depletion, i.e. whenNode connectivity values  were pre-calculated by the algorithm in advance, given the input network. Then NAGS and NFGS. In other words, lower NEA scores are then assigned to AGS-FGS pairs with more highly connected member nodes.An important feature of GNEA (gene-wise NEA) is that its enrichment estimates are, on average, based on fewer network edges compared to PWNEA, so that often https://cran.r-project.org/web/packages/NEArender/).The steps of NEA described above can be performed with functions available in R package NEArender . In order to get normally distributed values for our analyses, pPERT were transformed to Z-scores and signed according to the SPIA “Activated/inhibited” status as: Z.spia = qnorm*ifelse.The method by Tarca http://software.broadinstitute.org/gsea/msigdb/download_file.jsp?filePath=/resources/software/GSEA-P-R.1.0.zip . While GSEA possesses a sophisticated toolbox for significance estimation via permutation tests, we needed only the enrichment score and therefore calculated only the core ES values via function GSEA.EnrichmentScore. Normally, GSEA has been used for analyzing gene rankings from multi-sample analyses with replicates, such as a t-test of an experimental versus control group. The single-sample GSEA  needed for our analysis was described by Barbie et al.26. They produced sample-specific lists by ranking genes by absolute expression values in each given sample. We implemented this analysis under acronym ssGSEA. However this approach might miss sample specificity. As an example, such ubiquitously expressed genes as GAPDH, RPS16, and RPS11 were found among the top 10 items in more than 90% of the CCLE cell line transcriptomes. For this reason, we additionally implemented and tested ranking genes in each sample by z-scores, i.e. by the standardized deviations from the genes’ means across the whole cohort. Using this option, dubbed ZGSEA, was similar to mode topnorm for calculating AGS in function samples2ags of our package NEArender.The R implementation of GSEA was downloaded from The overrepresentation analysis, ORA estimated the significance of overlap between AGS and FGS in 2 × 2 tables. We did it via Fisher’s exact test using the function gsea.render in the R package NEArender described above. In order to get ORA values normally distributed, the “estimate” values from function fisher.test were augmented with a pseudo-score 0.1 and log-transformed.28 can combine results from twelve individual algorithms which were previously presented by third parties. EGSEA then calculates collective gene set scores for unified enrichment estimation. Since the framework of our analysis required assigning enrichment scores to individual samples rather than to multi-sample contrasts such as “experiment vs. control”, we excluded methods that were present in EGSEA but only applicable to multi-sample contrasts, namely Camera, Roast, Fry, PADOG, CAGE, SAFE, and globaltest. We used the remaining suitable methods ORA, ZSCORE, GSVA, PLAGE, and ssGSEA. The correlation analysis and other models in our work required input in the form of enrichment scores rather than p-values. Therefore we could combine scores with two EGSEA methods which were not based on p-values, namely min rank and average rank.The ensemble of genes set enrichment analysesF according to a general model of the form:ε denotes residual, i.e. unexplained by feature F, variance. The features were either original gene profiles from the three platforms  or scores from GSEA, or scores from the two NEA modes, PWNEA and GNEA, i.e. pathway-level network enrichment scores and single-gene network enrichment scores, respectively. All data sets, except the point mutation set, contained continuous variables and were thus analyzed using Spearman rank correlation. The point mutation data were analyzed using a one-way ANOVA model with two levels of F: “any mutation” versus “wild type”. P-values of both Spearman and ANOVA were adjusted by Benjamini and Hochberg method75.In each of the four drug screens, we quantified correlation between the cell line sensitivity to each drug and each of the molecular features Every tested model was built under 10-fold cross-validation using function cv.glmnet of R package glmnet  with the following parameters: lambda.min.ratio = 0.01 (the default) and nlambda = 25 (default was 100). Parameter alpha varied as {0.1; 0.3; 0.5; 0.9; 1.0}. The reported cross-validated mean error and the number of variables in the model corresponded to lambda.1se, i.e. largest value of lambda found within 1 standard error of the minimum lambda. The regression of observed on predicted values was plotted using lambda.min.th, 1/2nd, and full available (up to 18 years) intervals.We used the follow-up time profiles for which both status records “relapse/relapse-free” and “dead/alive” were available, which allowed creating “relapse-free survival” and “overall survival” variables. Depending on the cancer aggressiveness and chemotherapy type, different timeframes could become informative in the analysis of the eight TCGA cohorts. The follow-up timeframes were defined as 1/5in vitro screen and clinical data” we only considered 14 of the compounds, which were found in the in vitro sets. The p-values from this analysis were Bonferroni-adjusted in the cross-comparisons between the in vitro and clinical results.For the analysis reported in “Statistical power to detect correlates of drug sensitivity”, we used 42 drugs which were applied to at least 10 patients in one of the eight cohorts. In Fig. in vitro screens required accounting for multiple clinical variables. Such phenotype covariates as well as drug treatment data were obtained from TCGA as biotab files viaMatching significance of the drug-feature correlations that had been detected in the cell-line https://tcga-data.nci.nih.gov/tcgafiles/ftp_auth/distro_ftpusers/anonymous/tumor/*/bcr/biotab/clin/.C1…Ck and the residual term ε, main effects “drug” D and “feature” F as well as the interaction term D*F. A significant main effect of a drug could be interpreted as patients’ benefit in total and irrespective of the feature value, e.g. regardless of a gene mutation, or a gene expression, or a NEA-based pathway score. Conversely, a significant feature effect indicated that the feature correlated with survival directly, i.e. no matter if the drug was administered or not. Finally, significance of the interaction indicated efficacy of the drug specifically in patients with feature values either above or below a threshold, so that respective patterns could be explained by neither of the main effects. The interaction term was thus central for our purpose of detecting drug-feature correlations, whereas the significance of main effects of “feature” and “drug” was allowed although not required. As an example, a feature may or may not exhibit a significant correlation with survival in patients who did not receive the drug.In order to measure and probabilistically estimate these effects, we fitted Cox proportional hazards regression models for every feature versus drug combination. Using all covariates available a cohort  could result in unrealistically complex models. We thus included only covariates most likely associated with the disease prognosis, such as tumor degree, pathological tumor stage, immunohistochemical statuses in BRCA, Gleason score in PRAD, Karnofsky score in GBM . In order to estimate significance of the model terms, we used function coxph with continuous feature vectors. However, for visualizing the survival curves Fig.  each feaWe used the default parameters for function boxplot in R. The boxes contain data points within 25–75th percentile intervals (i.e. between quartiles Q1 and Q3). The maximal whisker length, MWL, is defined as 1.5 times the Q1–Q3 interquartile range (i.e. the box length). Whiskers can extend to either the MWL or the maximal available data point when the latter is below MWL. Markers thus correspond to data points that extend off the box by more than the MWL value.Supplementary Tables And FiguresDataset 1Dataset 2