Three aspects are highlighted in the PL21GPCA method. First, to degrade the high sensitivity to outliers and noise, the non-convex proximal Lp-norm (0 < p < 1)constraint is applied on the loss function. Second, to enhance the sparsity of gene expression in cancer samples, the L2,1-norm constraint is used on one of the regularization terms. Third, to retain the geometric structure of the data, we introduce the graph Laplacian regularization item to the PL21GPCA optimization model. Extensive experiments on five gene expression datasets, including one benchmark dataset, two single-cancer datasets from The Cancer Genome Atlas (TCGA), and two integrated datasets of multiple cancers from TCGA, are performed to validate the effectiveness of our method. The experimental results demonstrate that the PL21GPCA method performs better than many other methods in terms of tumor sample clustering. Additionally, this method is used to discover the gene network modules for the purpose of finding key genes that may be associated with some cancers.The dimensionality reduction method accompanied by different norm constraints plays an important role in mining useful information from large-scale gene expression data. In this article, a novel method named Lp-norm and L High-throughput sequencing technologies, including genome-wide measurements, have enabled large-scale gene expression profiles to accumulate faster . It is o0-norm constraint based on PCA to stress the sparse expression of genes in samples. The L1-norm  minimization  is the ith row of matrix X. 2,1-norm in the literature. To solve the L2,1-norm, we can compute the L2-norm of each row of X first, record it as a vector b(X) = , and then compute the L1-norm of vector b(X). The components of vector b indicate the importance of each feature. The L2,1-norm favors obtaining a small number of non-zero rows in matrix X, and then feature selection will be achieved.where X =  ∈ Rp×n (p≫n) be a matrix whose rows represent genes and columns represent samples. PCA is usually used to find the optimal principal directions VT =  ∈ Rk×n (VTV = I) that define the low-dimensional (k-dim) subspace. And the projected data points in the low subspace Vcan be denoted as the elements of the matrix Up×k =  ∈ Rp×k. The traditional PCA finds Uand V with the squared Frobenius norm:Let p (0 < p < 1)   is used  K-means . The optX =  ∈ Rp×n, xican be regarded as one data point (one vertex in the graph). For each data point xi, we find its k′ nearest neighbors and put edges between xi and its neighbors. Then, a graph with n vertices can be constructed, on which the weight matrix n×nW∈R is defined. wij is the weight between vertices xi and xj, it is used to measure the closeness of two points xi and xj, and it is a symmetric similarity matrix. There are three popular choices defining the weight matrix on the graph: heat kernel weighting, 0–1 weighting, and dot-product weighting. If nodes i and j are connected, using heat kernel weighting, wij = 1 using 0–1 weighting and Principal component analysis can find an approximate set of basis vectors in the case where data usually lie in a linear manifold . In consterature .ZT =  ∈ Rk×n represent the ndata points in the k-dim embedding coordinates ZT =  ∈ Rk×n (ZTV = I), i.e., the representation of x_i in the new low-dimensional basis is zi = . The “dissimilarity” of the two data points in the low basis can be measured by the Euclidean distance or the divergence distance. The Euclidean distance is adopted in our method. Define the “dissimilarity” of the two points in the low basis as d = ∥zi−zj∥2, combined with the weight matrix W, and the smoothness of the low-dimensional representation can be measured by minimizing:Let Tr(•)is the trace of a matrix, D=diagis a diagonal matrix, and L=D–W the Laplacian matrix p(0 < p < 1) is used on the loss function, which could improve the robustness of the optimization model effectively compared with the other constraints.To reduce the influence of outliers and noise, the non-convex proximal Lp-norm ∥g∥(2)2,1-norm is used on the projected matrix Up× k.To enhance the sparsity of gene expression in cancer samples, the L(3)To retain the intrinsic geometric structure of the data points (samples), the graph regularization item is recommended in the optimization model.X =  ∈ Rp×n(p≫n), which denotes p genes’ expression in n samples. Our goal is to find the optimal low-dimensional (k-dim) subspace denoted as ZT =  ∈ Rk×n (ZTV = I) and the projected matrix Up×k =  ∈ Rp×k in the low subspace. The traditional PCA finds Uand V with the squared Frobenius norm in the solution. In our optimization model, the proximal Lp-norm ∥g∥p(0 < p < 1)   replacesThe objective function of this method is designed as follows:Clearly, the objective function is somewhat intractable because it is non-convex and non-smooth. We adopt the augmented Lagrangian multiplier (ALM)  to addreX−UZT with E. Eq. (5) can be equivalently written as:When using the ALM method to obtain the optimal solution of (5), we replace According to the ALM method, eq. (6) is equivalent to minimizing:Y is the Lagrangian multiplier, and μ is the step size of the update rule. In (7), there are three variables to be solved. The alternating direction method (ADM) (where od (ADM)  is adoptU and V are fixed, (7) is written as follows:Problem 1: When where E is updated as:Let E and V are fixed, (7) is simplified as follows:Problem 2: When To simplify (11), let L with respect to Uare:The partial derivatives of p×pQ∈R is a diagonal matrix with U is then obtained:where i=1,⋯,p) . LettingTo simplify (14), let E and Ware fixed, (7) is simplified as follows:Problem 3: When With respect to the settings Vis found by minimizing:Based on (17), r+1V can be obtained as follows:Therefore, v1,…,vk) are the keigenvectors corresponding to the smallest k eigenvalues of the matrix k, p, λ, α, ρ, μ to be pre-determined, among them. As the parameters used to control the step size in the update rule of AML, we set μ = 10−2 and ρ = 1.2 for all gene expression datasets experiments , we choose them corresponding to different situations for the best clustering performance through extensive experiments. Different parameters are chosen for different datasets. Detailed parameter selections and discussions are described in section “Experiments and Discussion.”where (eriments . The park-dim) subspace denoted as T =∈Rk×n   (VT V=I)V, the classical clustering method K-means is then used for tumor clustering. For comparison, extensive experiments are also performed using existing dimensionality reduction methods, including SPCA  ∈ Rk×n (ZTV = I), the K-means algorithm is then employed for tumor sample clustering. The accuracy (ACC) and the normalized mutual information (NMI) are the two most commonly used metrics to evaluate the clustering results  equals 1 if x=y and 0 otherwise. The function map(ri) maps each obtained cluster label ri to the equivalent prior label. Let C be the prior set of clusters and C′ be the obtained set from our algorithm. Define their mutual information metric MI as:where p(ci) and ci and where H(C) and H(C′)are the entropies of C and C’, respectively. Therefore, the metric NMI ranges from 0 to 1. NMI=1 if the two sets of clusters are identical, and if the two sets are independent, NMI=0.where k, K-means was run 50 times on each dataset, and the average performance was computed. As a reference, we also recorded the maximum values of ACC and NMI of the 50 runs. Thus, four metrics, ACC_max, ACC_mean, NMI_max and NIM_mean, are used to evaluate our experiments. Generally, the larger the mean value is, the better is the clustering performance, and the better are the stability and robustness of the clustering. This also indicates that the corresponding dimension reduction method has good robustness and sparse effect.However, a problem that needs to be resolved is that the K-means algorithm may or may not converge to the same solution in each run with random initial conditions. Therefore, the evaluated metrics ACC and NMI obtained by only once-running of k-means is not enough to explain the result. To solve this problem, for the given cluster number p, λ, and α, which need to be determined in (5). The range of each parameter is 0 < p < 1, λ > 0, α > 0. When determining the optimal value of one parameter, the other two parameters are fixed. We focus on the influence of the value of p on the performance. PL21GPCA achieves consistently good performance when the two regularization parameters λ and α are varied from 10 to 1,000 on all three datasets. p at nine different values from 0.1 to 0.9. For every dataset, extensive experiments are carried out to seek the appropriate parameters to achieve the best performance for tumor sample clustering. Thus, different parameters are chosen for different datasets  on the loss function, employing the L2,1-norm regularization item to insure feature selection, and introducing the Laplacian regularization item to emphasize the geometrical structure of the data, the PL21GPCA-based method can always get a better clustering performance.For the different types of data used in the experiments, a number of meaningful points need to be emphasized further.For the lung cancer dataset, Finally, as we can see from To further show the performance of the novel dimensionality reduction method compared others, a visualized data distribution of the low-dimensional embedding corresponding to the first two components of the PCA-based method are demonstrated. Besides the proposed method PL21GPCA, the results of three other methods including SPCA, gLPCA, pgLPCA are compared because these methods are also the direct extensions of PCA. X3000×80 generated by rand function in Matlab. In order to simulate the representation of features in different types of samples, based on the generated matrix X3000×80, some changes have also been made. Firstly, we add 1 to the values of columns 1 to 20 in rows i∗30−29 of matrix X3000×80, add 2 to the values of columns 21 to 40 in rows i∗30−19, add 3 to the values of columns 41 to 60 in rows i∗30−9, add 4 to the values of columns 61 to 80 in rows i∗30−5, add 2 to the values of columns 21 to 40 in rows i∗30−25, add 1 to the values of columns 1 to 20 in rows i∗30−15, which means that the 80 samples in the simulation data contain four categories. Secondly, we use the function imnoise in matlab to add different sizes of Gaussian white noise to X. The mean value of the added Gaussian white noise is 0 and the variance 2σ is chosen in the range of [0.4∼1.2]. Next, we use the proposed method PL21GPCA and the compared methods to reduce the dimension and denoise the simulated data, and then use the K-means method to cluster the denoised data, the evaluation metric ACC_mean mentioned above is used to test the effectiveness of the method. the K-means algorithm is run 50 times to randomize the experiments.Experiments on simulation data are also carried out to evaluate the effectiveness of PL21GPCA. The simulation data used in the experiment is a matrix 2 = 0.4), with the increase of Gaussian white noise, the effect of our proposed method is mostly ahead of other methods especially when σ2 = 0.6, 0.8, 1.2, which shows that the new method has better de-noising ability and robustness.Due to the outstanding performance of our method on the CRC dataset and the integrated H_C_P dataset, the construction and analysis of the gene network are based on these two datasets. The strategy of gene network module discovery involves two steps. First, the genes for constructing the co-expression gene networks are selected. Second, based on the filtered genes, co-expression networks are established, and then the key genes that may be closely related to some cancers are analyzed.Up×k. Now, we mark the optimal projected matrix Up×k as In this step, there are two problems to be solved: one is how to select genes, and the other is how many to select. It is known that among thousands of genes, only a handful of them regulate a specific biological process . These mng to U~ . We denoThe upregulated genes are reflected by the positive value in the matrix ve value . Therefol(l≪p) elements. In many studies, it has been unclear how many genes should be selected for gene network analysis. Since only a small number of genes can regulate a specific biological process, these genes may play a decisive role in the clustering results of tumor samples. In this paper, the number of genes used for constructing the gene network is determined according to the clustering performance based on the selected genes. Through experimentally investigating the clustering performance with the number of selected genes varied from 500 to 2000, it is found that the clustering results corresponding to 1600 genes are best for the CRC data and 700 for the H_C_P data.The larger item in l differentially expressed genes are used to construct the gene network. Let matrix Rl×n denote the l gene expression in n samples. We use the Pearson correlation coefficient (PCC)   to measuhttp://www.genecards.org/, we list the annotations of the top ten genes in There are 39 modules, including 218 nodes and 504 edges, in the constructed network based on the CRC data. We analyzed the top 10 nodes (genes) with higher degrees in the first three modules that retained more relevant interactions. The degree of the node (gene) shows its role in the network modules. The larger the degree of the node (gene) is, the more important the node (gene) is, and such nodes (genes) may retain the tight connectivity of the network. The constructed network based on the integrated data H_C_P includes 157 nodes and 644 edges. We analyzed the five important nodes (genes) with higher degrees in the first three modules that retained more relevant interactions. p(0 < p < 1)is applied on the loss function to decrease the sensitivity to outliers and noise. The L2,1-norm is used on the projected matrix to enhance the sparse gene expression in cancer samples. The graph regularization item is introduced to the optimization model to retain the geometric structure of the data. Five gene expression datasets, including one benchmark dataset, two higher-dimensional single-cancer datasets from TCGA, and two integrated multicancer datasets from TCGA, are used to evaluate the performance of our method. The compared experiments demonstrate that the PL21GPCA method outperforms many existing methods in terms of tumor sample clustering. Moreover, this method is employed to discover gene network modules to find the key genes with close relationships to cancers. The results of our study may be a useful reference for clinical diagnosis.In this article, we propose a new dimensionality reduction method named PL21GPCA based on PCA for robust tumor sample clustering and gene network module discovery. Based on the traditional PCA, the non-convex proximal Lp-norm ∥g∥p(0 < p < 1), since Lp-norm minimization can result in a sparser solution than the L1-norm and perform better in terms of robustness to outliers than the L2-norm. However, in addition to the generalized shrinkage operation proposed by p < 1) minimization (p(0 < p < 1). Second, we will evaluate the performance of PL21GPCA as a compact representation method combined with other methods, including supervised and unsupervised clustering methods such as spectral clustering, support vector machine (SVM) or their improved versions. Third, as mentioned above, the PL21GPCA method gets especially outstanding performance for processing the integrated data, so we will use the PL21GPCA method to process many other integrated data to verify its performance further.There are some suggestions for future research. First, in the optimization model of PL21GPCA, the constraint used on the loss function is the non-convex proximal Lp-norm ∥g∥mization  problemshttp://www.unc.edu/∼haipeng. The TCGA data http://www.tcga.org/.Publicly available datasets were analyzed in this study. This data can be found here: The lung cancer data X-ZK conceived and designed the experiments. YS and X-ZK performed the experiments and contributed to the writing of the manuscript. S-SY, JW, and L-YD analyzed the data. J-XL and C-HZ contributed to reagents, materials, and analysis tools. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Applying the knowledge that methyltransferases and demethylases can modify adjacent cytosine-phosphorothioate-guanine (CpG) sites in the same DNA strand, we found that combining multiple CpGs into a single block may improve cancer diagnosis. However, survival prediction remains a challenge. In this study, we developed a pipeline named “stacked ensemble of machine learning models for methylation-correlated blocks” (EnMCB) that combined Cox regression, support vector regression (SVR), and elastic-net models to construct signatures based on DNA methylation-correlated blocks for lung adenocarcinoma (LUAD) survival prediction. We used methylation profiles from the Cancer Genome Atlas (TCGA) as the training set, and profiles from the Gene Expression Omnibus (GEO) as validation and testing sets. First, we partitioned the genome into blocks of tightly co-methylated CpG sites, which we termed methylation-correlated blocks (MCBs). After partitioning and feature selection, we observed different diagnostic capacities for predicting patient survival across the models. We combined the multiple models into a single stacking ensemble model. The stacking ensemble model based on the top-ranked block had the area under the receiver operating characteristic curve of 0.622 in the TCGA training set, 0.773 in the validation set, and 0.698 in the testing set. When stratified by clinicopathological risk factors, the risk score predicted by the top-ranked MCB was an independent prognostic factor. Our results showed that our pipeline was a reliable tool that may facilitate MCB selection and survival prediction. Lung adenocarcinoma (LUAD) is one of the leading causes of cancer-related death . The pooObtaining reliable and quantitative measurements using the minimum number of markers is challenging, and more sensitive assays need to be developed. Despite recent progress, the molecular mechanisms underlying prognosis have not been explained in detail. In the search for new potential human cancer biomarkers, the hypermethylation of cytosine-phosphorothioate-guanine site (CpG) island sequences located in the promoter regions of tumor suppressor genes is gaining prominence . SeveralThese MCBs have the potential to substantially improve the accuracy of diagnosis prediction . PreviouThere is a demand for an advanced model that can identify patterns underlying CpG changes in such MCBs and disease progression, as methylation changes are correlated with disease-free survival (DFS) . HoweverIn this study, we presented a novel pipeline named stacked ensemble of machine learning models for methylation-correlated blocks (EnMCB) that automatically finds methylation-correlated blocks and introduces them as signatures to build machine learning models . We builWe analyzed and developed a methylation signature using Illumina Infinium Human Methylation 450K BeadChip data obtained from the Cancer Genome Atlas (TCGA) and Gene Expression Omnibus (GEO). We extracted clinical information on the TCGA data from the Pan-Cancer Clinical Data Resource .http://gdac.broadinstitute.org/). The GSE39279 methylation profiles  and a testing set (2/3). The clinical characteristics of the samples in the two public datasets are summarized in To determine the DFS of LUAD profiles in TCGA, we defined a disease event as when a patient has a new tumor event, including local recurrence, distant metastasis, or new primary tumors at all sites. DFS was defined as the time before a disease event, measured from the date of surgery to the date of the disease event. Patients were designated as censored cases when their days to last contact met the criteria. Since new primary tumor events are not released in Methylation profiles were preprocessed using a chip description file. This chip description file, which includes CpG-based annotation files from the Illumina Infinium Human Methylation 450 K BeadChip, allows for direct mapping of CpGs to the corresponding genes (Ensembl IDs or gene symbols). These annotation files allowed us to extend gene annotation and labeling by providing a table that contains the gene symbols and other CpG characteristics in an R expression set.β value, which we calculated as: Briefly, the degree of DNA methylation at each CpG was denoted as a yi,methy and yi,unmethy are normalized values of the methylated and unmethylated allele intensities, and ε is the constant offset  that was recommended by Illumina. There were a total of 485,577 CpG features. The beta value ranged from 0 to 1, reflecting the fraction of methylated alleles at each CpG in each sample.where We removed the CpG sites missing values or that were associated with known single-nucleotide polymorphism (SNP) loci, and replaced the CpG sites with a few missing values (<30%) with the mean value of the whole feature. A total of 391,513 features remained after preprocessing.β values were processed using noob background correction  between any two adjacent markers that indicated uncorrelated methylation. Markers not separated by a boundary were combined into the MCB.We evaluated a genome-wide DNA methylation profile of 455 samples. To avoid any potential heterogeneity, we excluded 32 adjacent normal lung samples, two samples with no primary tumors, and three replicated samples from the original cohort of 492 samples in the LUAD TCGA database. All samples in the training set were used to partition the genome into blocks of tightly co-methylated CpG sites (MCBs) according to . We calcThis procedure identified a total of 31,726 MCBs, with two to 45 CpG positions in each block. We also calculated the Pearson correlation coefficients between the two adjacent CpGs and the standard deviations.The relative gene expressions were calculated using the RNAseq transcriptome data from TCGA database. Because of the wide variation of raw count values, we log2 transformed the values. We then identified genes with methylation values that correlated with varied associated gene expression levels using a Pearson correlation test. If the MCB covered multiple gene transcription start sites, we selected the gene that had the correlation coefficient with the greatest absolute value as the corresponding gene for this MCB. If no correlated gene was found, we set the result as “not available” (NA).LUAD tumor samples were selected from patients with available DFS clinical data (not NA). We selected 455 methylation profiles from a total of 492 samples in TCGA, and a total of 155 out of 444 samples using the same criterion from the GEO dataset. We next assessed the prognostic utility of the methylation signatures for survival based on the selected data and using the ensemble model with two stages, which are described in 2.6.1 and 2.6.2, respectively.For individual CpGs, we used univariate Cox proportional hazards regression to evaluate the independent prognostic factors (CpGs) associated with DFS. To enable quantitative analysis of the methylation patterns within individual MCBs across many samples, we needed a single metric to define the methylated pattern of multiple CpG sites within each block.λ = 0.01) based on Cox regression with partial likelihood deviances to assess the correlation of survival and to filter out unrelated MCBs. We collected the results for filtered MCBs to show their prediction capacity.We calculated the arithmetic mean of all CpGs in each MCB, and used those arithmetic mean values for feature selection. We selected MCBs with at least five intro CpGs. Then, using the MCB arithmetic mean values, we carried out the embedded feature selection step  with L1 Apart from the arithmetic mean, we constructed three separate learning models as follows:GCOX. The minimizing coefficients were defined using the ordinary least squares method. Second, we built the support vector regression GSVR model using a linear kernel function (details can be found in GEN (λ) on the predictive model.First, we constructed the classic Cox regression model d in GEN  was consThe prediction values for all three models in the training set were determined using the 10-fold cross validation method. For predictions in the testing set, we used the median score in the training to generate the cutoff signature score. The parameters used for tuning the models, package information, and environment details are recorded in All CpG risk scores were calculated in individual MCBs using Cox, SVR, and elastic-net models. We used the predictions as the compound methylation MCB values. We then constructed multi-model-based stacking ensemble classifiers  to predigi based on all the samples x ∈ 𝔛 with the formula Seeking a blended prediction function wi is a learned model weight in ℝ. In feature-weighted linear stacking, f1, f2, …, fj represent a collection of j meta-feature functions to be used for stacking. The weight values wi can be modeled as linear functions of the meta-features where vij is the learned weights. where vij can be estimated using the training set We vat optimized the problem by minimizing the loss function in Finally, we calculated the risk score for each patient based on the individual MCB models. The prediction values for the ensemble model in the training set were also determined using the 10-fold cross validation method.We then tested the univariate associations of the clinical pathological characteristics  and eachi = 1, …, n, represented each MCB, and j represented each model, i.e., the results of the area under the receiver operating characteristic (ROC) curve (AUC) that tested the associations between RFS and Cox, SVR, Elastic-Net and ensemble model predictions in the training and validation sets. Next, we ranked the summary scores of the prediction AUCs for each model, which formed Rij = rank(AUC). Small rank values were marked for better results.where We used Cox proportional hazards analysis to evaluate the performance of the associations between the models and the DFS clinic factor. We further evaluated the performance of the models using ROC curves, followed by calculating the AUC  and C-int-test to compare classifiers over multiple datasets (MCBs)  in Bioconductor .We defined the cut-off value for the survival curve as the median value of the risk scores in the training set. We used bootstrap percentile method  to compas (MCBs) . We calc. method . A p valonductor . The soun = 455) to identify the methylation correlation blocks. Because of the underlying assumption that DNA sites in close proximity are co-methylated, we studied the degree of co-methylation across different DNA strands by using the well-established concept of genetic linkage disequilibrium. We applied a Pearson correlation method (using an r2 cutoff of 0.8) to quantify the co-methylation of close CpG sites. Then, we partitioned the genome into blocks of tightly co-methylated CpG sites.We used the DNA methylation microarrays within all primary LUAD , which showed low β values, intermediate β values, and high β values.β values across multiple CpG sites within one MCB, but we also observed a relatively high standard deviation for the mean β values of multiple CpG sites in individual MCBs. In some MCBs, the standard deviation was as high as 0.28  in the training set (The prognostic capacities (AUC) of the three MCB-based models, namely elastic-net regression (AUC 0.386–0.717), SVR (AUC 0.359- 0.658), and Cox regression modeling (AUC 0.389–0.729), are shown in ning set  when comλ = 0.01, we preserved a total of 297 MCBs (We next performed embedded feature selection (L1 Regularization) based on Cox regression with the MCB arithmetic mean. Using a threshold of 297 MCBs .p > 0.05, paired t-test) when compared to the ensemble model. The median AUCs in the training set were 0.505, 0.520, 0.515, and 0.505 for the Cox, SVR, elastic-net regression, and ensemble models, respectively. The maximum AUCs in the training set were 0.649, 0.607, 0.616, and 0.649, for the Cox, SVR, elastic-net regression, and ensemble models, respectively. The box plots for the MCB AUCs are shown in After feature selection, we calculated the signature score for individual MCBs in separate models  using 10-fold cross validation. We assayed the AUC and C-index for each MCB in the training set . Our resPSD3 gene (contained 5 CpGs) had AUCs of 0.622 in the training set, 0.773 in the validation set, and 0.698 in the testing set  to assess their performance in the training and validation sets. After training and validation, the models were selected and finalized. The holdout cohort (103 samples) was used as testing set and only for final validation. We identified the MCBs with the top five RP values in the training and validation sets . The MCBting set .We also carried out another procedure that combined the TCGA and the GEO datasets. We randomly divided those mixed samples into the training and the testing sets (8:2). The MCBs were identified and selected by L1 penalty using the training set. We further analyzed the performances (AUCs) of multiple models using 10-fold cross validation in the training set. We found that the results using cross validation (training/testing) procedure were similar to that of the training/validation/testing method. The MCB-29016-based classifier was also top ranked (only based on the training set) and show good results in the testing set. The AUC in the training set (AUC 0.658) was slightly higher than that of in the testing set (AUC 0.650). The results are shown in p > 0.05, bootstrap percentile method).We used time-dependent ROC analysis with 5-year follow-up times for the training and testing sets to assess the prognostic accuracy of the MCB-29016-based classifier  and 6B. p values in the training sets  in pathological stages 1–2, tumor-node-metastasis (TNM) stages T1-T2  can be read at the bottom lines. For more details (nomogram usage) please read . The MCBs were identified and selected by the L1 penalty using the training set. We further analyzed the performances (AUCs) of multiple models using 10-fold cross validation in the training set. We found that the results using cross validation (training/testing) procedure were similar to that of the training/validation/testing method. MCB-29016-based classifiers were also top ranked (only based on the training set) and show good results in the testing set. The AUC in the training set (AUC 0.658) was slightly higher than that of in the testing set (AUC 0.650). The ranking was based on the AUC results of MCB-based classifiers in the training and testing sets.Click here for additional data file.10.7717/peerj.10884/supp-8Table S5Click here for additional data file.10.7717/peerj.10884/supp-9Table S6Click here for additional data file.


Major depressive disorder (MDD) is a global health challenge that impacts the quality of patients’ lives severely. The disorder can manifest in many forms with different combinations of symptoms, which makes its clinical diagnosis difficult. Robust biomarkers are greatly needed to improve diagnosis and to understand the etiology of the disease. The main purpose of this study was to create a predictive model for MDD diagnosis based on peripheral blood transcriptomes.We collected nine RNA expression datasets for MDD patients and healthy samples from the Gene Expression Omnibus database. After a series of quality control and heterogeneity tests, 302 samples from six studies were deemed suitable for the study. R package “MetaOmics” was applied for systematic meta-analysis of genome-wide expression data. Receiver operating characteristic (ROC) curve analysis was used to evaluate the diagnostic effectiveness of individual genes. To obtain a better diagnostic model, we also adopted the support vector machine (SVM), random forest (RF), k-nearest neighbors (kNN), and naive Bayesian (NB) tools for modeling, with the RF method being used for feature selection.AKR1C3, ARG1, KLRB1, MAFG, TPST1, and WWC3) with a false discovery rate (FDR) < 0.05 between MDD patients and control subjects. We then evaluated the diagnostic ability of these genes individually. With single gene prediction, we achieved a corresponding area under the curve (AUC) value of 0.63 ± 0.04, 0.67 ± 0.07, 0.70 ± 0.11, 0.64 ± 0.08, 0.68 ± 0.07, and 0.62 ± 0.09, respectively, for these genes. Next, we constructed the classifiers of SVM, RF, kNN, and NB with an AUC of 0.84 ± 0.09, 0.81 ± 0.10, 0.73 ± 0.11, and 0.83 ± 0.09, respectively, in validation datasets, suggesting that the SVM classifier might be superior for constructing an MDD diagnostic model. The final SVM classifier including 70 feature genes was capable of distinguishing MDD samples from healthy controls and yielded an AUC of 0.78 in an independent dataset.Our analysis revealed six differentially expressed genes (This study provides new insights into potential biomarkers through meta-analysis of GEO data. Constructing different machine learning models based on these biomarkers could be a valuable approach for diagnosing MDD in clinical practice. From 1990 to 2016, major depressive disorder (MDD) was one of the five leading causes of years lived with disability . PatientEarly diagnosis and appropriate treatment would undoubtedly reduce the incidence and mortality rate of MDD patients. However, like many other affective disorders, the complex etiology of MDD and the inevitable need for clinical judgment based on an individual’s medical history may cause a lack of reliability in diagnosis. More objective diagnostic methods thus are required. Previous studies have explored molecular biomarkers of MDD based on genomic, epigenetic, transcriptomic, and proteomic sources . SeveralBDNF) had inconsistent results. The studies of This study focused on the peripheral transcriptomic biomarkers, which have been described as “sentinels of disease” . Becausein vitro macrophage models to perform a meta-analysis and identified consistently differentially expressed genes (DEGs) that have been implicated in inflammatory and metabolic processes. Therefore, in this study, meta-analysis was first performed to identify consistent biomarkers from different large-sample datasets. Although there are various meta-analyses of microarray data, they generally focus on one or a few genes; few have been developed for systematic integration of multiple microarray datasets . The curTFRC (transferrin receptor) can predict recurrent MDD with an accuracy of 63%.MDD is influenced by both genetics and environment where the transcriptome feature patterns or feature function patterns may represent disease subtypes, outcome prognosis, drug benefit prediction, or specific biological process. The machine learning  approachThe aim of this study was to identify potential transcriptional biosignatures that might be used for the diagnosis of MDD. Here, we applied meta-analysis to discover DEGs differing between MDD patients and healthy controls. Six significant DEGs with FDRs < 0.05 were investigated for their diagnostic capability. To obtain better diagnostic efficacy, we compared four ML models. Finally, an SVM prediction model consisting of 70 feature genes was constructed and validated by a reserved independent gene expression dataset.1. Then we conducted a systematic search in the GEO repository2 using the following search sentence: ((((((((MDD) OR major depressive disorders) OR depressive disorders) OR depressive syndromes) OR depression)) AND (((blood) OR peripheral blood) OR PB)) AND Homo sapiens [Organism]) AND Expression profiling by array [Filter]. A report was included in the analysis if the following criteria were satisfied: (1) used a case-control design; (2) patients did not have any diseases other than MDD; and (3) the patients were medication free. We finally obtained a total of 9 datasets  librarydatasets .Nine microarray datasets were retrieved from the GEO database: GSE98793, GSE19738, GSE38206, GSE52790, GSE39653, GSE76826, GSE58430, GSE32280, and GSE46743, with a sample size of 128, 67, 18, 22, 45, 22, 12, 16, and 160, respectively .3 were used to normalize the data and annotate the probe information. For the GSE19738, GSE38206, GSE58430, and GSE76826 data, based on the Agilent platform , the quantile method was used to normalize the data. Annotation of the probe information was based on Agilent platform information. For the GSE39653 and GSE46743 datasets based on the Illumina platform , the quantile method was used to normalize the data, and annotation of the probe information was based on Illumina platform information.For the GSE98793, GSE52790, and GSE32280 datasets, based on the Affymetrix platform , the raw CEL data were downloaded; and the Robust Multi-Array Average (RMA) method and the ‘‘Oligo’’ package from BioConductor4 for initial processing. We excluded those genes with more than 50% of the data missing. The most variable probe measured by inter-quartile range (IQR) was used to handle redundant probe sets that correspond to the same gene.Normalized signal intensity data were imported into BRB-Array Tools (v. 4.5)Meta-analysis of microarray data was carried out in “MetaOmics” based on R language , which ip-value from formal hypothesis testing in each QC criterion , external consistency of co-expression pattern with pathway database (EQC), accuracy of biomarker detection (AQCg), accuracy of enriched pathway detection (AQCp), consistency of differentially expressed genes (CQCg), and consistency of enriched pathway ranking (CQCp). Each QC index was defined as the minus log-transformed riterion . Finallyriterion .MetaDE was used for identifying differentially expressed genes, and the meta-analysis method used in the current study was developed by 5; 6) to construct the FI network  was applied in the following steps.p < 0.01 were considered potential biomarkers for further feature selection. Then, the dataset GSE98793 with the largest sample size was used as the training dataset in both feature screening and ML modeling. The other four datasets in discovery sets were used for internal validation.The six datasets included in this study were divided into two parts: GSE98793 , GSE1973We applied an RF algorithm to reduce the number of feature genes, with the following steps: (1) ranked genes in descending order based on their importance; (2) eliminated feature genes one by one according to the importance of each feature with the goal of producing a new feature set; and (3) repeated the above process with the new feature set. We used 10-fold cross-validation for verification and calculated the average accuracy value to assess the classification capability. Finally, the feature set with the highest average accuracy was selected for model construction.The following ML algorithms, SVM , RF Tin, kNN Al, and NB The aim of SVM algorithm is to identify a decision hyperplane that make the distance between the hyperplane and the instances that are closest to boundary is maximized. By introducing the concept of “soft margin” and using “kernel trick,” SVM performs well with linear indivisibility data . SVM witRandom Forest is an ensembles-learning algorithm forming with a series of decision trees. Each tree is developed from a subset from the training data. The class of the new instance is determined by using the majority vote of individual trees in the forest . In thisIn kNN, classification is based on the distance between the instances and an object is classified according to the status of its k nearest neighbors. R package “kknn” was used in the modeling, where we set distance = 2 (Euclidean Distance), and tuned the parameter “kmax.”Naive Bayes is a statistical classification algorithm based on theBayes theorem. The core idea of Naive Bayes is, if the probability ofinstance x belonging to A is greater than the probability of belonging to B under some attribute conditions, it is said that instance x belongs to A. R package “klaR” was used in the modeling and output parameter “usekernel.”Default values were used for other parameters in each model. To evaluate the overall performance of each model, leave-one-out cross-validation was performed. Details about the tested parameters and their corresponding test values for each model are provided in We used the average AUC to assess the classification capability of each model. Finally, a model with the highest average AUC in validation sets was chosen.To facilitate clinical application, we attempted to construct a model with fewer genes without affecting the accuracy of classification efficiency. We compared the classification ability of the model based on the average AUC value of discovery set. The criterion for determining the final model was that the model achieved the optimal average AUC value in the discovery sets.The feature genes in the final determined SVM classifier were used to perform the supervised clustering of samples and extents of expression. The clustering results were visualized using a heatmap .The information in the datasets is shown inp < 0.01. Of them, 66 were upregulated and 71 downregulated in MDD (TPST1), arginase 1 (ARG1), killer cell lectin-like receptor B1 (KLRB1), WWC family member 3 (WWC3), aldo-keto reductase family 1 member C3 (AKR1C3), and MAF bZIP transcription factor G (MAFG). The forest plots of the six genes’ expression in different datasets are shown in The quality of the eight datasets was assessed utilizing “MetaQC.” Among the eight microarray datasets, six were included in the further meta-analysis for DEGs , and thed in MDD . A detaiMLKL, CEP63, and CSNK1E, a moderate negative correlation among DNAJC7, MLKL, and CSNK1E , which plays an important role in interconversion of androgens, estrogens, and progestins to their cognate inactive metabolites  can be found in the article/SZ, ZB, XZ, and MX collected the data, conducted the analysis, and drafted the manuscript. ML provided resources and revised the manuscript. ZY administered project and revised the manuscript. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


We propose a metric called condition-number based deviation, or cdev, to quantify normalization success. cdev measures how much an expression matrix differs from another. If a ground truth normalization is given, cdev can then be used to evaluate the performance of normalizers. To establish experimental ground truth, we compiled an extensive set of public RNA-seq assays with external spike-ins. This data collection, together with cdev, provides a valuable toolset for benchmarking new and existing normalization methods.Normalization of RNA-seq data has been an active area of research since the problem was first recognized a decade ago. Despite the active development of new normalizers, their performance measures have been given little attention. To evaluate normalizers, researchers have been relying on  As a prominent application of RNA-seq data, DE analysis has been routinely used to assess normalization quality  data. The problem has been studied for a decade, resulting in a large array of methods . Despite quality . Picking quality  is mostl quality . Intuiti quality . Other m quality . Similar quality . Along t quality  was used quality . Althoug quality .via traditional measures such as false positive rate, false negative rate, false discovery rate, area under the ROC curve, etc., and their ability to recover true expression levels, via the mean squared error (MSE) of log-ratios, either for all genes  genes  genes . Experime) genes . Indeed,e) genes , making e) genes . Specifie) genes . Such gre) genes  which mue) genes  and caute) genes  the use e) genes . In regre) genes . In facte) genes , Lastly,e) genes  which glThe complicated landscape of performance metrics for RNA-seq normalization, as briefly surveyed above, exhibits several issues. First, comparative studies heavily depended on DE detection, an analysis downstream of normalization. In the early days of RNA-seq assays, normalization and DE detection are almost always coupled. As RNA-seq data are used in a wider range of applications, it becomes insufficient to evaluate a normalizer only in terms of DE analysis performance. Moreover, it is not practical to assess the performance of a normalizer on all possible downstream applications. Second, most performance measures in the field are data-driven. A major caveat with these metrics is the ability to optimize them directly, resulting in algorithms which will easily top the rank if judged by their own objective functions. For example, an algorithm could be devised to optimize the de-correlation of gene expression profiles , or to mcdev, which measures the differences between two expression matrices. The cdev between a normalized expression matrix and the ground truth is then a succinct indicator of normalization performance. cdev was first utilized by cdev is indeed concordant with normalization quality when measured by both upstream and downstream criteria. In companion with cdev, we compiled a collection of spiked RNA-seq assays which can serve as experimental ground truth for normalization. This toolset will assist in the validation and comparison of RNA-seq normalizers, independent from downstream analyses and unbiased toward any particular generative model of RNA-seq read counts.In addressing these issues, we propose a ground-truth based measure called condition-number based deviation, or n genes in m samples of two conditions, according to an empirical distribution obtained from real data  for up-regulated genes, or Unif for down-regulated genes, such that 50% of the DE genes are up-regulated, and 50% are down-regulated. This choice is arbitrary, since the distribution of fold-change in real data sets comes in all shapes and sizes, and even shifted by normalization.Simulation of RNA-seq counts was performed in the same manner as done by eal data . In m/2 A summary of different simulation scenarios were given in RNA-seq data were collected from the Gene Expression Omnibus (GEO) database based on the following criteria: having ERCC spike-ins added at a constant concentration across a sufficiently large number of samples. The list of studies where these assays were originally performed is available in All RNA-seq assays from said studies were downloaded from the Sequence Read Archive (SRA) in FASTQ format and re-processed by a single workflow. In this workflow, the raw reads were aligned to the corresponding transcriptome using STAR v2.7 , and quan genes on m samples typically results in an n × m matrix Y, the element yij of which captures the measurement of gene i in sample j. In RNA-seq, Y is the read count matrix output by the quantification process. To enable between-sample comparison, the read counts in Y requires proper normalization. The normalized expression matrix X is obtained by scaling Y against a set of m factors ν = {ν1, ν2, …, νm} such that every gene in sample j is scaled by the same factor νj. Let R be the set of reference genes whose expression levels are stable across all samples. If this set is identified, the scaling factor νj can be computed for each sample as followsAn expression profiling experiment of ν by a scalar results in a valid normalization. To obtain expression levels at a comparable magnitude with the raw counts, we chose to scale these factors by their geometric mean. This step facilitates the visual comparison of expression matrices but does not critically affect the outcome of differential expression analysis  between X and A, denoted cdev. To compute cdev, we first define the transformation from X to A. Let B be the matrix that transforms X to A, i.e.,XB = A. As we are interested in recovering the pattern of differential expression rather than the absolute concentration of each RNA species, any multiplication of the ground truth A by a scalar is considered a valid normalization. Equivalently, X is considered valid if B = αI, where α is any scalar. Hence the quality of X can be measured by how much B deviates from the identity form. Such deviation is quantified by the condition number of B, defined as κ(B) =  ∥ B ∥ ⋅ ∥ B−1 ∥. Choosing ∥• ∥ to be ℓ2 norm, the condition number can be calculated using the ratio between the largest and smallest singular values, that is, B, one may recognize that B = (XTX)−1XTA, in which (XTX)−1XT is the Moore–Penrose pseudoinverse of X, denoted as X†. Hence the definition of cdev can be written as followsX† can be computed using the singular value decomposition X = UΣVT, that is, X† = VΣ−1U  depends on that of the pseudoinverse X† and the singular value decomposition of X†A, both of which are guaranteed to exist. Hence cdev always exists for any expression matrices X and A. As the singular values of X†A are unique, so is cdev.Symmetry. One should note that, cdev is not symmetric, i.e., cdev ≠ cdev for generally given matrices X and A. However, when the transformation from X to A involves only column-scaling, we have cdev = cdev. To see why, we first notice that the transformation matrix B, by which XB = A, is now diagonal,A to X is then C = B−1. By definition of cdev,cdev is a symmetric measure.Interpretation. cdev ranges from 1 to infinity, with a smaller cdev indicating more similar expression patterns. The definition of cdev is closely related to that of a valid normalization. Specifically, any multiplication X′ of the matrix X by a scalar results in cdev = 1.Y by the full set of references R, denoted XR, should be similar to that of scaling by the subset S ⊂ R, denoted XS. Hence, cdev should be close to 1. This implication is helpful in validating the behavior of cdev on real data sets (See the Results section).Because any non-empty subset of references result in a valid normalization, the result of scaling Implementation. Although cdev can be computed for any X and A of the same shape using the definition above, it is much more trivial and efficient to compute in practice, when X is transformed to (or from) A by global scaling. In particular, the singular values of diagonal matrix B are now its elements. These elements also define the normalization vector v which can be re-constructed by dividing an arbitrary row X of X by the corresponding row A of A, as long as the feature i is not zero in any sample.cdev, we compared six common normalization methods, including Total Count (TC), Upper Quartile (UQ) (As an illustration for the use of ile (UQ) , trimmedile (UQ) , DESeq (ile (UQ) , DEGES/Tile (UQ) , and Poiile (UQ) . These mm-vector v, the element vj of which is used to scale sample j. While some methods produce close-to-1 scaling factors, some others require further normalization of these factors, for example, by the geometric mean of the elements vj, to keep the magnitude comparable. As the magnitude of v does not affect the validity of its resulting normalization, a simplified description of v resulted by each method is provided below to highlight the differences between them.Each of these methods produce an vj is proportional to the total count of all genes measured in sample j, i.e.,In TC, the scaling factor In UQ, the factor is computed by the upper quartile instead of the summation over gene counts, G∗ is the set of genes with reads in at least one sample.in which j and the reference sample, arbitrarily selected as the first one, In TMM, the scaling factor is computed from the ratio of a gene count in sample GTMM denotes the set of genes retained by TMM criteria.in which j and that in the pseudo-reference sample, which is computed by taking the geometric mean of each gene over the samples, In DESeq, scaling factors are computed from the medians of the ratio between a count in sample t is computed using the same method as that of TMM, while restricting the set of genes to that of Gt, DEGES is an iterative strategy in which the count matrix is scaled by a normalization method , DE genes are identified by a DE caller , and the normalization is repeated with DE genes now removed . There aGt is the set of genes identified as non-DE at iteration t.in which PoissonSeq is also an iterative protocol which starts with fitting a Poisson log linear model under the null hypothesis that no gene is DE, then removes those with low ranking for goodness-of-fit, and fits again under the new hypothesis until convergence. The scaling factor is computed in the similar manner to that of TC, Gt is the set of genes remained at iteration t.in which limma in consideration of its speed and robust performance relative to other methods  analysis was used as a downstream indicator of normalization quality. Among a great number of methods developed for calling DE genes, we chose  methods . The mets , log-transformed, and normalized, using the function  package . The resprotocol .limma-trend, DESeq . Earlier studies such as Some earlier observations were reproduced on these data sets. First, the read counts of ERCC spike-ins are highly correlated with one another, especially those at high and moderate levels. This phenomenon was consistently observed in all data sets , corroboWithin each data set, the samples were spiked with the same concentration of ERCC spike-in RNAs and prepared with the same experimental protocol. Consequently, spike-ins are the true references in each data set. After normalizing by these references (see Methods section for the exact formula), the levels of ERCC spike-ins become well equalized  curve (AUC)  . When fohreshold –2F. Whenhreshold –2J.cdev is agnostic to the magnitude of scaling factors, as are DE gene calling procedures. This behavior is demonstrated empirically on simulated data sets. DE performance is almost identical on expression matrices scaled by the same-direction vectors spanning several orders of magnitude. Not only limma, but also count-based methods such as DESeq and edgeR, were not affected by the magnitude of scaling factors  should be close to 1 with A being normalized by the true reference set R, and A′ by any non-empty subset S of R. Meanwhile, cdev where Arand is resulted from a random set of genes should be larger. These expectations were observed in all the real data sets , cdev is larger than cdev, but never exceeds 10. In highly different samples, such as those from 16 fly strains , random normalization results in very large deviation from the ground truth, with cdev mostly around 100. Data sets with average heterogeneity, such as those of rat tissues  or sarcoma samples (sarcoma), exhibit cdev in the middle range.Since the behavior of ata sets . In highcdev in practice, at the same time implies the importance of normalization with respect to the heterogeneity of transcriptomic data. Specifically, when the expression profiles are highly similar, wrong normalization is not as detrimental as it would be on heterogeneous ones. Based on the behavior of cdev on real data sets , cdev up to 10 may be acceptable for DE analysis. On moderately diverse expression profiles , cdev below 5 may be necessary to obtain good enough DE calling. On closely related expression profiles , cdev should be as low as 2 in order to provide a good enough DE results. As cdev is a continuous measure of normalization quality, these values should not be treated as hard threshold, but rather a guideline to understand the normalization outcome. The intent of cdev is not to predict downstream analysis performance, but rather to enable the measurement of normalization quality independent from downstream analyses, as would be argued in the Discussion section.These observations illustrate the value range of ata sets , we may i.e., non-DE) genes. The latter version requires less information from the ground truth and works equally well as the former, hence chosen to compare with cdev. When defined for true references, the expected fold-changes are all one.Among a few ground-truth based measures of normalization quality, fold-change MSE is an intuitive metric. As the name implies, a fold-change MSE closer to zero indicates a better normalization. In cdev and fold-change MSE with respect to the corresponding ground truth. To ensure these randomly normalized matrices cover a wide enough range of quality, we varied the fraction of differential genes allowed in the set of scaling genes. On this simulated data, the two metrics are highly correlated , while the true ones properly removed such variation. Although there may seem to be a few instances with “high” fold-change MSE and “low” cdev, they are in fact part of a cdev-MSE function with a lower slope, thus do not contribute to the discordance between the two metrics. Inspection of such cases confirm that these random normalizations are clearly worse than the true one  are iterative procedures in which the set of non-differentially expressed features are refined until convergence. In all the following evaluation, the ground truth is computed by scaling against the set of known references. For simulated data, this set includes the genes with fold-change set to 1 by the simulator, such that the number of DE (and non-DE) genes satisfies the required percentage of differential expression. For real data, it includes selected ERCC spike-ins which have sufficiently high number of reads. In both simulation and real data, the true references were able to equalize themselves across all samples  genes  genes . As the es /2 pairwise comparisons to make, hence the number of fold-change MSE to compute. In contrast, cdev is grouping-agnostic and capable of comparing expression matrices as a whole. As demonstrated in the Results section, cdev is also more robust than fold-change MSE, especially on heterogeneous conditions.Once a ground truth is defined, performance can be measured, conveniently and objectively, by a similarity (or dissimilarity) measure. Although MSE is an intuitive and easy-to-compute metric, the definition of fold-change MSE on more than two conditions becomes overly cumbersome. An experiment with cdev could in principal be computed for any pair of expression matrices X and Y, it is not as meaningful outside the context of global scaling normalization. Since cdev is only symmetric when the transformation from X to Y (and Y to X) involves only per-sample scaling factors, operations other than that will require additional convention about the direction. However, we would advise against such convention and recommend to keep cdev limited to the evaluation of global-scaling normalization. It is a necessary restriction, for these operations are not necessarily mutually exclusive with other non global-scaling ones, such as the removal of batch effects (Although  effects , or per- effects . For exa effects . Due to de facto measures are either qualitative or potentially subjected to biases. Some were introduced without much justification. We elaborated on the definition of cdev, its properties and applications, showing that this metric is highly indicative for normalization performance, proportional to both upstream processing and downstream analyses in simulations. Along with this metric, we re-processed and curated a collection of RNA-seq data sets with ERCC spike-ins. We envision this toolset will be highly valuable in developing and evaluating normalization methods.Performance measure has long been in the shadow of RNA-seq normalization, despite a vibrant scene of normalizing method development. Most of the 10.7717/peerj.12233/supp-1Supplemental Information 1Click here for additional data file.


Patient stratification based on drug response prediction is hence needed for effective anti-cancer therapy. However, single-gene markers of response are rare and/or may fail to achieve a significant impact in the clinic. Machine Learning (ML) is emerging as a particularly promising complementary approach to precision oncology. (2) Methods: Here we leverage comprehensive Patient-Derived Xenograft (PDX) pharmacogenomic data sets with dimensionality-reducing ML algorithms with this purpose. (3) Results: Combining multiple gene alterations via ML leads to better discrimination between sensitive and resistant PDXs in 19 of the 26 analysed cases. Highly predictive ML models employing concise gene lists were found for three cases: paclitaxel (breast cancer), binimetinib (breast cancer) and cetuximab . Interestingly, each of these multi-gene ML models identifies some treatment-responsive PDXs not harbouring the best actionable mutation for that case. Thus, ML multi-gene predictors generally have much fewer false negatives than the corresponding single-gene marker. (4) Conclusions: As PDXs often recapitulate clinical outcomes, these results suggest that many more patients could benefit from precision oncology if ML algorithms were also applied to existing clinical pharmacogenomics data, especially those algorithms generating classifiers combining data-selected gene alterations. It is now well-established that the efficacy of cancer drugs is strongly patient-dependent. Whereas analgesics, such as Cox-2 inhibitors, show efficacy in 80% of patients, on average, only 25% of oncological patients actually respond to cancer drugs . ConsequNot only are these simple drug-gene associations rare, but they are also not strong predictors of drug response in some cases. For example, the mutational status of EGFR in Non-Small Cell Lung Cancer (NSCLC) is an FDA-approved marker of response to Erlotinib ,8. The rWhile precision and recall vary depending on the drug and its actionable mutation, the example in A promising complement to single-gene markers is indeed the application of Machine Learning (ML)  to learnUnfortunately, a major limiting factor for the application of ML to this problem is the availability of relevant data   models are important when no relevant clinical data is available ,24,25,26Here we carry out a systematic study to investigate how ML can improve the prediction of in vivo drug response from tumour molecular profiles by combining multiple gene alterations. To the best of our knowledge, we were the first to analyse NIBR-PDXE data with this aim. Note that previous studies applying ML to this problem across multiple drugs have been based on pharmacogenomics data from in vitro cell lines, instead of PDXs. Three additional aspects of this paper are novel: (1) directly comparing the performance of ML classifiers against that of single-gene markers across multiple treatments, (2) introducing and applying a variant of ML classifier intended to provide a more stringent Feature Selection (FS)  as a wayGiven that single-gene markers based on somatic mutation data are central to current genomic medicine ,35, it iOne major challenge to build any predictive model is the high dimensionality of pharmacogenomics data. While typically, only tens of tumours have their response to the drug available, the molecular profiles of these tumours may easily aggregate over 50,000 features. To face this challenge, ML algorithms with built-in FS, such as Elastic Nets ,38,39,40Lastly, the application of an ML method achieving slightly better performance than a previous method on average across drugs is commonly reported. However, ML methods can perform similarly on average over a set of drugs but dissimilarly on one of those drugs. Therefore, we adopt a non-competitive approach instead, where the goal is to identify the most suitable profile and classifier type for each treatment. This approach should be collectively more predictive than using the method with the highest average performance across treatments. Another expected advantage is that we will be able to find out which molecular profile is most predictive for a given drug, which is valuable to reduce the time and cost that would be associated with determining the rest of the profiles on patients treated with that drug.https://static-content.springer.com/esm/art%3A10.1038%2Fnm.3954/MediaObjects/41591_2015_BFnm3954_MOESM10_ESM.xlsx (accessed on 25 September 2021). This Excel file has five tabs named RNASeq_fpkm, copy_number, pdxe_mut_and_cn2, PCT_raw_data and PCT_curve_metrics. The first three tabs contain three molecular profiles of the xenografted tumours. The RNASeq_fpkm tab contains gene expression values. The copy number tab contains the actual copy number of each gene. Copy number is also available as a categorical variable at the pdxe_mut_and_cn2 tab . Around 400 PDX models were profiled at each of these omic levels. The other two tabs are for treatment response data. The raw response data tab (PCT_raw_data) includes the percentage of tumour volume change relative to tumour volume at the start of treatment (%∆TVol) of each treated PDX recorded every 3–4 days. Lastly, the processed response data tab (PCT_curve_metrics) includes the categorisation of PDX responses into one of four classes calculated from raw response data. Further information about how this data set was generated can be found in the original study [The NIBR-PDXE data set  is publial study .For each treated PDX, we retrieved its category from the processed response data. We also calculated its category from raw response data as indicated by Gao et al. . Retrievpdxe_mut_and_cn2 tab). If no SNV was detected in this gene, the gene was labelled as Wild Type (WT), and the SNV feature was assigned a value of zero. This encoding scheme is commonly used [copy_number tab contains the actual copy number (CN) of each gene as determined by Gao et al. [pdxe_mut_and_cn2). As previously performed in [RNASeq_fpkm Table. Thus, each PDX profiled at the GEX level is characterised by 22,665 real-valued features (one per gene as well).For each gene, the Single-Nucleotide Variant (SNV) feature for that gene is assigned a value of one if at least one SNV was detected in this gene region ) state the numbers of sensitive and resistant PDXs per treatment for BRCA and CRC, respectively.Only a part of the PDX models from NIBR-PDXE have been both treatment-response and molecularly profiled. A previous cancer pharmacogenomics modelling study showed that it is possible to predict the drug response of held-out tumours with an ML model trained on just 35 tumours . As we aNote that no claim about the optimality of the 35PDXs threshold is made. Although less likely, some cases with fewer PDXs might be predictive too. In any case, a threshold is required to make the analysis manageable. We run 2 ML algorithms for 2 cancer types (BRCA and CRC), 13 treatments per cancer type, 4 types of molecular profiles for each treatment-cancer type pair and 10 replicates. Thus, a total of 2080 Leave-One-Out (LOO) Cross-Validations (CV) runs have been carried out. A less restrictive threshold would further increase the number of trained and tested ML models, as well as the subsequent analysis, which is already extensive for a single study.ith treatment can be represented as:ith treatment has been administered to in PDX models. In these binary classification problems, positive data instances are PDXs sensitive to the considered treatment (class = sensitive), whereas negatives are resistant PDXs (class = resistant). Note that, while they have slightly different meanings, we use the terms responder and sensitive PDX interchangeably as it is customary (same applies to the terms non-responder and resistant PDX).The pharmacogenomics data set for a given cancer type, molecular profile, and the Each of these data sets is employed to train classifiers to predict the class of a PDX from its corresponding molecular profile. Classifiers not employing model selection  are evaluated with standard LOOCV, whereas those employing model selections are evaluated with nested LOOCV to avoid overestimating their performance ,52. It iOnce observed classes are compared to predicted classes, PDXs in the considered data set can be broken down into true positives, true negatives, false positives and false negatives . Thus, the discrimination offered by a classifier can be summarised by the Matthews Correlation Coefficient (MCC) .(2)MCC=MCC can take values from −1 to 1, where 1 means that the classifier provides a perfect agreement between the observed and predicted classes, −1 indicates a perfect disagreement and 0 means that the classifier performance is equivalent to that of predicting the class at random.To investigate how the two sources of error contribute to the overall predictive performance represented by MCC, we also calculate Precision (PR) and Recall (RC) for each predictor. PR and RC are two classical metrics  whose deIn this study, a PR value of 0 would mean that all the PDXs predicted sensitive by the classifier are actually resistant, whereas a PR value of 1 would indicate that all the PDXs predicted responsive were experimentally confirmed to be responsive. On the other hand, RC is 0 when none of the sensitive PDXs is correctly identified as such, whereas RC is 1 if no sensitive PDX is missed by the classifier. An alternative global metric capturing both types of errors is the F-score (F1), which is the equally-weighted harmonic mean of PR and RC. The optimal value of F1 is 1 , whereas its worst value is 0 .The Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) were calculated using the R package ROCR version 1.0–7. The ROC curve was built based on the response class predicted by the model (RF-OMC or SG) and the actual observed class of each test PDX. For a given model, the ROC curve shows all different pairs of values for the metrics True Positive Rate (TPR) and False Positive Rate (FPR) resulting from varying the operating threshold from 0 to 1 to cut-off the probability of the PDX to be a positive (responsive). RF-based models assign each test PDX with a class probability value between 0 and 1; thus, their ROC curves have one point for each threshold leading to a different TPR-FPR pair. By contrast, SG markers predict the class of each test PDX as either resistant/non-responsive (0) or sensitive/responsive (1); thus, all thresholds between 0 and 1 lead to the same TPR-FPR pair . The AUC is the area under the ROC curve. AUC ranges from 0 to 1. AUC = 1 denotes a perfect classification, whereas AUC = 0.5 corresponds to a random-level prediction of classes.try   is one oproblems ). We preproblems ,58,59,60data_by_treatment_BRCA and data_by_treatment_CRC in the The proportion of responsive and non-responsive PDXs changes from case to case .p-values from two-sided Fisher’s exact tests to rank binary features within a given binary profile (SNV or CNA) and p-values from two-sided unpaired t-tests to rank real-valued features (GEX or CN). Each p-value measures how strongly that feature discriminates between responsive and non-responsive PDXs. For each profile, treatment and cancer type, we considered n/2 subsets of features (n is the number of PDXs available for that case): the subset with the top 2 features, that with the top 3 features, …, that with top n/2 features and finally all features for that profile. This limit of n/2 ensures that the ML algorithm will not be challenged by high-dimensional data , except for the run using all features intended to find out whether the case requires more than the top n/2 features. Lastly, the best among these n/2 models is selected as that with the highest LOOCV MCC. To estimate its performance, nested LOOCV MCC is calculated with model selection in the inner loop [However, the optimal number of features and their identities depend on various factors . Consequently, we designed OMC as a strategy to build ML models, employing only the most relevant features. In a nutshell, OMC is made of three modules: one to rank features according to their relevance to treatment response, another to train an ML model per considered subset of features and a third one to select the optimal model among those trained. Regarding ranking features, we used ner loop ,52 (the t-test to determine whether the performance of the RF model was better than that of this random model (p < 0.05). Results can be found in The following protocol was followed for each case. First, the proportion of sensitive PDXs in each LOOCV training fold was taken as an estimate of the probability of a PDX being sensitive. Second, a random number between 0 and 1 was generated according to the estimated prior probability to decide whether the held out PDX in the LOOCV test fold was predicted as sensitive or not. Third, the process was iterated over all LOOCV folds. Once the response class of every PDX had been predicted in this way, a single MCC value was computed from predicted and actual classes of the PDXs. Lastly, we repeated this process 10 times with the same set of 10 different random seeds employed with RF-OMC and RF-all. From the 10 MCCs for the case and the other 10 MCCs for the RF-model , we carried out a one-sided paired p-values and effect sizes  was identified. The operation was repeated for each fold resulting in a LOOCV predicted class for each treated PDX and thus the LOOCV MCC for the best marker. After evaluating its predictive performance, we recalculated each best single-gene marker, now using all the data so that these markers are ready to be used on forthcoming tumours. These results are in the single-gene_marker tab).We identified the best single-gene marker for each of the 26 treatment-cancer type pairs using exactly the same data as RF-all and RF-OMC via LOOCV. The SNV profile was used as the source of detected somatic mutations, as there is currently a strong interest in using them as pharmacogenomic markers in oncology . For eacsizes (ϕ ) for 15,We started the analysis by determining which drug-cancer type pairs in NIBR-PDXE  have sufp-value and reported its LOOCV performance. Full results can be found in the “single-gene_markers” tab of the To perform this task, we trained and evaluated two ML algorithms on each data set using LOOCVs, as detailed in the Methods section. The first algorithm is RF using all available features , whereas the second is an OMC variant of RF to identify the most predictive features in each case (RF-OMC). Both algorithms are used to build binary classification models. To account for their stochastic nature, we trained each algorithm on each LOOCV training fold 10 times, thus obtaining 10 estimations of each performance metric per case. Note that this study always reports the median performances of each algorithm on held-out PDXs that were not used to train or select the model providing the prediction . To assess which cases are better predicted by a particular gene, we also performed the standard single-gene analysis to evaluate which genes sensitise PDXs to the treatment when an actionable SNV is detected. In particular, we identified the sensitive marker with the lowest p = 0.47 from a one-sided paired t-test, both algorithms obtaining an average MCC of 0.19). However, RF-OMC was found to outperform RF-all in BRCA . The best predictor in the remaining seven cases, mostly in CRC, was a single-gene marker based on SNV data . Highly significant p-values strongly suggest that the corresponding lists of selected features have a high predictive value. By contrast, p < 0.05; one-sided paired t-test).For further validation, we have also compared each of these two ML classifiers against a random model where the response class is predicted using the prior probability of a PDX to be sensitive. Each of the models in this study is evaluated using independent data by LOOCV . CV is a standard way to assess predictive accuracy not only in this topic ,62,63,64In the next three subsections, we describe the three predictors found to have the highest accuracies in more detail. These three are the most useful models to predict which forthcoming PDXs will be responsive. The best multi-gene predictor for BRCA PDXs treated with binimetinib, a MEK1/2 inhibitor, was RF-OMC applied to GEX data comprising the expression values of 22,665 genes. Our analysis discarded the other three molecular profiles for binimetinib-BRCA , as they were substantially less predictive than the GEX profile in this case . The multi-gene predictor achieves a more substantial discrimination between sensitive and resistant PDXs than the PABPC3 marker. This is also indicated by a higher MCC (0.57 vs. 0.24). To help to understand what MCC values represent in terms of achieved discrimination, we also indicated FP and FN errors from both classifiers.The best multi-gene predictor for BRCA PDXs treated with paclitaxel was RF-OMC applied to somatic mutation data comprising the presence or absence of an SNV in 15,232 genes. Our analysis discarded the other three molecular profiles for paclitaxel-BRCA , for being less predictive than the SNV profile in this case , its performance in left-out PDXs suggests that it is a spurious correlation.The best multi-gene predictor for CRC PDXs treated with cetuximab was RF-OMC applied to somatic mutation data. The other three molecular profiles for cetuximab-CRC , were discarded for being less predictive than the SNV profile in this case . Unlike RF-OMC, most prediction errors correspond to sensitive PDXs that were not correctly identified as such (FN = 7 vs. FP = 1).In the three cases analysed in In the three cases that do not follow this trend, a slightly higher proportion of correctly predicted sensitive PDXs was found using single-gene markers. The first case is BGJ398-BRCA, and its best single-gene marker is the drug-gene association BGJ398-KIF20B. The other two cases are in CRC and have as best markers the following drug-gene associations: BYL719-TMEM184A and BYL719+LJM716-LSR.To shed light on this question, we identified the treatment-cancer type pair whose RF-OMC model obtains the median LOOCV MCC across the 26 pairs, which is BYL719—CRC—CN with an MCC of 0.24. Instead of only using CN features, we merged all the profiles and ran RF-OMC again on this much larger set of features . The MCC of this model was 0.04, and hence performance degraded in this case by adding many more features.p-value = 0.003, two-sided paired t-test).For each treatment-cancer type pair, we identified the profile with the best RF-OMC model and the number of top features selected by OMC. Then we selected at random from this profile the same number of features. Lastly, the LOOCV of an RF model using these randomly selected features was carried out 10 times, each time with a different random seed. p-value is calculated for each feature according to how well the feature individually discriminates between the two classes (sensitive and resistant). Then features are ranked from the most discriminative to the least discriminative using these p-values. Critical features are those that are found to be the most discriminative, and the OMC simply decides the cut-off . We could have selected another way to rank features according to how discriminative or predictive they are. For example, one where the correlations between features is taken into account. This is the case of the feature importance tool of RF, where each feature is noised up to measure the resulting decrease in accuracy of the RF model. Thus, the correlation of the noised-up feature with other features influences the ranking of the features by how predictive they are, at the cost of having to run RF with all features to estimate their importance. While investigating this additional algorithm is out of scope, we have illustrated this point in the case of one of the 102 datasets  is employed instead of the standard RF algorithm see . An impoIn terms of performance, the most predictive multi-gene ML model achieved an MCC of 0.57 . We are not aware of similar studies using PDX data, but there are a number of them exploiting cell line data. Thus, it is possible to put this level of predictive accuracy in the context of other test set performances reported in the literature for this problem. One study  applied How was this level of performance achieved? Considering multiple types of classifiers and molecular profiles collectively improved the prediction of in vivo tumour response across treatments and cancer types. We observed that combining multiple gene alterations via ML results in better discrimination between sensitive and resistant PDXs in 19 of the 26 analysed cases. More importantly, this ML approach has determined the most predictive classifier type and molecular profile for each treatment—cancer type pair. Despite training on data from practically the same PDXs within a cancer type, we found that some treatments can be predicted much better than others (this is true for both cancer types). The results show that increasing the number of considered classifiers leads to higher average accuracy across treatments , with only the highest-ranked being used to train the ML model. Thus, such univariate filters may miss cooperativity effects among features. In principle, wrapper FS techniques, such as Recursive Feature Elimination (RFE) [We have seen that even ML algorithms with built-in FS can often struggle to provide predictive classification models. This is due to overfitting caused by training on high-dimensional data ,74, and on (RFE) , may impon (RFE) , univarion (RFE) , a novelIn practice, we have found that OMC complements the standard version of RF on this type of problem (16 of the 26 cases were better predicted by RF-OMC). More importantly, RF-OMC predicts 11 cases with an MCC of at least 0.3, whereas RF-all only predicts 4 cases with at least that accuracy see . FurtherThe results also show that the accuracy in predicting in vivo tumour response strongly depends on the treatment, regardless of cancer type. The boxplots in As anticipating which classifiers will work best on a case is currently not possible, these are benchmarked on a similar case to select those most suitable . Here weWhile the generalisation of these methodology research outcomes to patient data is still to be confirmed, the fact that they have been observed in both in vitro and in vivo preclinical models is promising. If such generalisation was observed, without generating any additional data, many more patients would benefit from precision oncology by applying multiple ML algorithms to existing clinical pharmacogenomics data . Moreover, as PDXs capture the diversity and complexity of their originating tumours, the translational potential of the generated predictors in a clinical setting is likely. Improved predictors for paclitaxel and cetuximab, both of which are standards of care for breast and colon cancers, respectively, could hence have an impact on cancer treatment effectiveness in the clinic. Our approach could also be applied to generating improved predictors of response to drugs in development , supporting the use of ML for patient selection in clinical trials. Beyond these potential clinical applications, our study shows that ML can be employed to improve our understanding of cancer biology. On the one hand, ML could anticipate which PDXs not harbouring the actionable mutation are responsive (these would have therefore been missed by existing single-gene markers). On the other hand, the OMC strategy provides a concise list of gene alterations that control treatment response in the considered cancer type  can determine the most predictive classifier type and molecular profile for each treatment- cancer type pair. Thus, by evaluating several models  and molecular profile types , the prediction of a given case can be effectively increased. Moreover, the ML classifiers also obtain higher recall than the corresponding SG markers; therefore, many more patients could benefit from precision oncology by applying complementarily multiple ML algorithms than from using only the current SG markers.Another advantage of RF-OMC over the standard RF model is that this OMC version requires only a concise list of genes to be profiled to predict the sensitivity of a PDX to a cancer drug. This gene list permits cheaper and faster clinical implementation. Furthermore, it can be considered as a promising starting point for further studies in the underlying mechanisms of drug sensitivity in cancer cells. Last but not least, the most predictive RF-OMC models revealed the genes that are not just data-driven but biologically relevant to cancer, as found by other independent experimental studies.