Topic models are algorithms introduced for discovering hidden topics or latent variables in large, unstructured text corpora. Leveraging on analogies between texts and gene expression profiles, these algorithms can be used to find structures in expression data. This work presents an application of topic modeling techniques for the identification of breast cancer subtypes. In particular, we extended a specific class of topic models to allow a multiomics approach. As an illustrative example, considering both messenger RNA and microRNA expression levels, we were able to clearly distinguish healthy from tumor samples as well as the different breast cancer subtypes. The integration of different layers of information is crucial for the observed classification accuracy. Our approach naturally provides the genes and the microRNAs associated to the specific topics that are used for sample organization. We show that indeed these topics often contain genes involved in breast cancer development and are associated to different survival probabilities.The integration of transcriptional data with other layers of information, such as the post-transcriptional regulation mediated by microRNAs, can be crucial to identify the driver genes and the subtypes of complex and heterogeneous diseases such as cancer. This paper presents an approach based on topic modeling to accomplish this integration task. More specifically, we show how an algorithm based on a hierarchical version of stochastic block modeling can be naturally extended to integrate any combination of ’omics data. We test this approach on breast cancer samples from the TCGA database, integrating data on messenger RNA, microRNAs, and copy number variations. We show that the inclusion of the microRNA layer significantly improves the accuracy of subtype classification. Moreover, some of the hidden structures or “topics” that the algorithm extracts actually correspond to genes and microRNAs involved in breast cancer development and are associated to the survival probability. A crucial problem in modern computational biology is the integration of different sources of information in the framework of the so-called “precision medicine” . Thanks In particular, topic modeling has been recently proposed as a computational technique to identify hidden structures in gene expression data ,3. TopicLDA is based on the assumption of a Dirichlet prior for the latent variables. This choice simplifies the statistical inference problem making the algorithm highly efficient. However, many complex systems in which LDA is applied, including expression data, are characterized by the emergence of power-law distributions, which are very far from the Dirichlet assumption ,9,10,11.To overcome these problems, a new class of algorithms based on hierarchical Stochastic Block Modeling (hSBM) was recently proposed . These aWe recently performed a comparative study  of diffeThis paper extends our previous study by integrating in the hSBM framework multiple layers of information. While the integration of additional biological information should generally improve the accuracy of the statistical inference, it is important to stress that this is not always trivially true. Highly noisy or irrelevant data layers could interfere with the task. We will show an empirical example of such a negative interference. Therefore, the addition of new layers should be driven by a clear biological motivation.nSBM, inherited from hSBM [First, we will show how different layers of biological information can be efficiently integrated in the hSBM framework. We release the python package rom hSBM , which iSecond, focusing on breast cancer, we will show that the combination of microRNA and protein-coding expression levels greatly improves the algorithm’s ability to identify cancer subtypes. These findings further confirm the important role previously recognized in several studies that miRNAs play in cancer development ,20.Third, we use the inferred topic structure to select a few genes, miRNAs, and chromosomal duplications that seem to have a prognostic role in breast cancer and, thus, could be introduced as additional signatures of specific breast cancer subtypes. The extension of subtype signatures can help clinicians to fine-tune diagnostic protocols in the framework of a precision medicine approach to cancer .We will focus on the illustrative case of breast cancer, which is the most commonly diagnosed cancer type and the leading cause of cancer death in women worldwide , with thMany real-word networks are accompanied by annotations or metadata describing different node properties. For example, in social networks, information about age, gender, or ethnicity can be associated to the nodes or the data capacity can be associated to the nodes of the Internet network . In a sin types: sample nodes , and  and, analogously, the links connecting to miRNAs report the miRNA expression level. When we add a layer with the CNV information, the links are weighted with the number of copies of the gene in the connected sample. The algorithm interprets the weight Once the multipartite network is defined, the statistical inference procedure leading to the topic structure is a straightforward extension of the procedure developed for the hierarchical Stochastic Block Model (hSBM) , which wThe model uses a generative process to build a network given a set of parameters  A.  and distinct (there are no blocks containing different kind of nodes). However, given its probabilistic nature, the algorithm can be naturally extended to fuzzy clusters.Lack of a parametric prior.Thanks to the network-based approach and to the particular way links are used to update the block structure, this class of algorithms does not require a specific parametric assumption for the prior probability distribution of the latent variables. This is a major difference with respect to LDA and makes this class of algorithms particularly suited for biological systems in which long-tail distributions and hierarchical structures are ubiquitous (see the discussion on this point and the comparison with LDA in ).Probability distributions over latent variables of different types.The output of the algorithm is not deterministic but is instead a set of probabilities that associate a sample with latent variables of different types As we will show in the following, these probability distributions capture relevant properties of the biological system.Hierarchical topic structure.Blocks and the probability distributions described above are available at different layers of resolution, from few large sets (clusters/gene-topics/miRNA-topics) at low resolution to many small sets at a higher resolution. The specific number of layers and their block composition are found by the algorithm optimization process and are not given as input. Therefore, the datasets can be organized in different ways depending on the resolution of interest. Note that not all possible resolutions are trivially present, as in standard hierarchical clustering.Concurrent and separate topic organization of the different network layers.Different ’omics have typically different normalization, and the numbers associated to different molecular features have often a very different meaning. A major advantage of nSBM with respect to other algorithms ,27 is thThere are several features that distinguish hSBM, and its nSBM extension introduced here, from other clustering or topic modeling algorithms such as LDA.The benchmark task we now focus on to test the performance of nSBM is its ability to cluster breast cancer samples according to their subtype annotation. This is an important task for its clinical relevance, but also because the breast cancer subtype could be dependent on a complex combination of factors, including gene and miRNA expression profiles; thus, the classification could be a good test for nSBM.Breast cancer is indeed a heterogeneous disease, with wide variations in tumor morphology, molecular characteristics, and clinical response ,28,29,30Luminal A, Luminal B, Triple-Negative/Basal, HER2, and Normal-like. For our tests, we used as a benchmark the TCGABiolinks annotations [Breast cancer samples are usually divided into five different subtypes: otations ,33, as dotations . The agrWe recently demonstrated that topic-modeling-based algorithms can achieve satisfactory performances in this classification task by looking at gene expression profiles   can further improve the hSBM results presented in ref. .It is, by now, well-established that miRNAs play an important role in several human diseases, particularly in cancer. Accordingly, miRNAs have been proposed as diagnostic biomarkers of human cancers ,36,37. TFollowing this line of evidence, we integrated miRNA expression levels with protein-coding mRNA levels using a We first tested if the integration of miRNAs has an effect on the partition of samples in clusters and on the topic organization in the gene branch.Therefore, the addition of the miRNA branch can radically affect the inferred topic structure and the clustering of samples.We first tested the ability of the algorithm in recognizing healthy from cancer samples. The hSBM algorithm showed good performances on this task by considering only gene expression data , as summThe two model settings (hSBM and triSBM) are compared quantitatively in As a second benchmark, we tested the ability of triSBM to identify breast cancer subtypes. Again, triSBM and hSBM are compared and the results are reported in Subtype Selected labels provided by TCGABiolinks [We used the Biolinks ,33 as thNote that the standard characterization of breast cancer subtypes relies on the expression level of only few markers. We did not explicitly select these markers in our gene selection process; thus, as previously discussed , the emeGiven these positive results, we will explore in the following sections the biological information contained in the latent variables inferred by the algorithm and test their possible prognostic role.We compared the blocks we obtained in output with the annotations of TCGA sample in . First oWe measured the Normalized Mutual Information score of both the bipartite (hSBM) model and the model that integrates miRNA (triSBM). Results are reported in The overlap between our cluster partition and two independent nonoverlapping labels can be explained by the fact that our partition groups samples at the intersection between the two labeling systems.We applied the same pipeline applied on TCGA to the METABRIC  dataset A major advantage of a topic modeling approach to multiomics data is that we can use the information stored in the probability distributions lysis of , we consEquation  of the MThese topics are nothing but lists of genes and can be investigated using a standard enrichment analysis. The results shown in this paper were computed using the Gene Set Enrichment Analysis GSEA  tool. InWe discuss the results of this analysis in the following two subsections.We report in There are topics, such as, for instance, topic 8 in Another interesting pattern is well-exemplified by topics 27, 28, and 44 in Looking at the figures and at the table, we see a few interesting patterns.While the above results were similar to the ones already discussed in , the novhttps://github.com/BioPhys-Turin/keywordTCGA/blob/main/brca/trisbm/trisbm_level_0_topics.csv, accessed on 10 February 2022) is the typical example of a topic that shows no particular preference for a cancer subtype  activatory role by interfering with a repressed epigenetic pathway. These are the so-called “epi-miRNAs” ,60 that To integrate the regulatory information, we made use of the analogy of this problem with inclusion of the citation information among documents in standard topic modeling applications to texts . In our We ran the tripartite model as described before; then, in a second moment, we added links gene–miRNA from regulatory network . It is well-known that chromosomal aberrations are a hallmark of cancer and that several types of cancer are characterized by a well-defined set of chromosomal loci whose deletion or duplication can drive the onset of that particular type of cancer. We already noticed that, using the information contained in the miRNA branch, we could identify two loci whose alteration were known to be associated with the onset of breast cancer. In TCGA database, we also have the information on the CNV values for all samples. We included this information by adding a fourth branch to our algorithm . As a preliminary test, we selected only genes with positive CNV  and that were neglected for the moment deletions.We performed a gene selection also in this new branch. Highly Copied Genes were selected, keeping the ones with an average (over samples) CNV greater than It is important to stress that, at this stage nodes, which corresponds with the same gene in the gene expression branch and in the CNV branch, are completely uncorrelated and are seen by the algorithm as independent nodes. We shall discuss below how to address this issue.In our setting, we have We ran the tetraSBM model on this network with samples, protein-coding genes, miRNAs, and CNV genes and obtained two hierarchical levels. In the first one, the four branches were partitioned into 13 clusters, 7 gene-topics, 5 miRNA-topics, and 5 CNV-topics. In the second one, we found 397 clusters, 49 gene-topics, 14 miRNA-topics, and 31 CNV-topics.Looking at the CNV-topics, we found a very interesting result see . Performdecrease in the NMI value  managed by the NCI and NHGRI. Information about TCGA can be found at Subtype_Selected. Those subtype annotations are provided by [In order to benchmark our results, we compared in vided by  and are vided by ,67 aboutsample type 10 or 11 in TCGA barcode  .In other analyses, we needed to know if a sample was a primary tumor or derived from normal tissues. Solid Normal Tissues samples are the ones with We downloaded the independent Breast Cancer Consensus Subtypes (BCCS) related to the TCGA files provided by the Supplementary files of .We downloaded METABRIC data from the European Genome-Phenome archive.We downloaded METABRIC miRNA landscape study (EGAS00000000122), in particular, Normalized miRNA expression data (EGAD00010000438) and Normalized mRNA expression (EGAD00010000434).The search for optimal allocation of the latent variables is performed by inheriting and expanding  hierarchminimise_nested_blockmodel_dl function from graph-tool [t from group r to group s is [t and s; t. From this, another advantage of a multibranch approach should be clear: different ’omics may have their own normalization. In fact, when moving a sample from r to s, the probability is estimated considering only the branch to which t belongs. If the node t is a gene, As mentioned in the main text, nSBM attempts to maximize the posterior probability rametric  way. Insaph-tool . In our oup s is (2)Pr→s|We set the algorithm so as to do a sort of model selection minimizing the Description Length We used the nested, degree-corrected  version V is the number of vertices  and E is the number of edges. If E. We ran the model on a 48-core machine with 768 GB of memory [The intrinsic complexity of typical Stochastic Block Modeling algorithms is (E∼O(V)) , where Vf memory .We collect here some further information on the nSBM algorithm.In our setting, we have We considered 1222 samples from TCGA-BRCA project and selected the ion from ,33; thenAs the output of the model, we find the probability distributions The same is true for miRNA-topics and for each and every eventual additional layer of features.The data provided in the atlas consisted of We performed two kinds of preprocessing to reduce the number of nodes and the number of edges.scanpy python package [In order to reduce the number of nodes, we filtered genes and miRNAs selecting only the highly variable ones. The highly variable are the ones with the highest dispersion (variance over mean) with respect to the genes with the same average expression. This selection was performed using the  package . This anFurthermore, we applied a standard approach to reduce the weights of the links and applied a In the Copy Number Variation analyses, we chose An interesting feature of the SBM type of algorithm is that they are typically robust with respect to gene selection. In the analyses of this paper, we considered only highly variable genes; however, in the supplementary material of , we discIn the analysis of the METABRIC dataset, we utilized the previously selected genes and microRNA.To evaluate the agreement between the sample partitions and the annotations, we chose the so-called “Normalized Mutual Information” (NMI), which was proposed in , in a neNMI/NMI∗, which measures how much information the model learns with respect to a random assignment. It is important to stress that this measure has no absolute value and should not be used to compare performances on different datasets; however, it can be successfully used to compare different algorithm in the same dataset, which is what we did in the Results section.The E is the total number of edges), it is possible to measure the quantity of information that the model requires to describe an edge. In the models of In addition to the NMI, it is also possible to compare different classes of topic modeling algorithms on their ability to compress the data ,74. Thiss factor  (5)P¯, and ended up with four stages: i, ii, iii, and iv.Our analysis began with the list of the mixtures demographic.vital_status, demographic.days_to_last_follow_up, demographic.days_to_death, demographic.gender, and diagnoses.age_at_diagnosis. We estimated the lifetime or the number of days the patient survived after the diagnosis, using days_to_last_follow_up if the patient was Alive and days_to_death for Dead patients. A similar approach was recently utilized by [Using Genomic Data Commons tools, we downloaded TCGA metadata and, in particular, lized by .In order to estimate whether a topic is upregulated in a patient, we evaluated the 35th percentile of COX model. These analyses were performed using the lifelines Python package [We used these data to fit the hazard with a  package  and, in n covariates considered. The hazard is defined as the ratio of the derivative of the survival and the survival itself t, namely, the number of patients alive at time t divided by the total number of patients. The package estimated the ratio between the hazard of samples with topic upregulated and hazard of samples with topic not upregulated. Therefore, we were able to estimate the coef does not depend on time but is a sort of weighted average of period-specific hazard ratios.The Cox model quantified how the topic of miRNAs regulation affected the survival probability. Cox fits the hazard function conditioned to a variable https://github.com/BioPhys-Turin/nsbm, accessed on 10 February 2022) or, alternatively, can be installed using Anaconda  by running conda install nsbm -c conda-forge.The Python package to run nSBM  can be dWe discussed in this paper the application using genomics data; however, the package is written in a way that makes it agnostic with respect to the type of data it receives in input and to the number of branches. One can ideally integrate as many different sources (’omics) of data as needed. Eventually, it can process not only biological data, but every kind of dataset whose input could be represented as a rectangular matrix (Bag of Words) for each feature.nSBM, inherited from hSBM [Using the python package: rom hSBM , ready tSecond, the integration of multiple sources of data, such as microRNA expression levels and the protein-coding mRNA ones, greatly improves the ability of the algorithm to identify breast cancer subtypes.Third, we use our results to identify a few genes and miRNAs and characterize a few chromosomal duplications that seem to have a particular prognostic role in breast cancer and could be used as signatures to predict the particular breast cancer subtypes.In conclusion, the nSBM model we propose here, integrating multiple sources of information into an hSBM analysis, should be useful to extract a lot of information from transcriptomics data.In conclusion, this paper released a new tool to easily integrate different sources of data into a topic-modeling analysis.We showed some application in a specific case (breast cancer) with some sources of data . Indeed, this approach can be applied to other datasets and, more importantly, to any possible sources of data .


In 2013–2014, Israel accelerated adoption of activity-based payments to hospitals. While the effects of such payments on patient length of stay (LoS) have been examined in several countries, there have been few analyses of incentive effects in the Israeli context of capped reimbursements and stretched resources.We examined administrative data from the Israel Ministry of Health for 14 procedures from 2005 to 2016 in all not-for-profit hospitals (97% of the acute care beds). Survival analyses using a Weibull distribution allowed us to examine the non-negative and right-skewed data. We opted for a Bayesian approach to estimate relative change in LoS.LoS declined in 7 of 14 procedures analyzed, notably, in 6 out of 7 urological procedures. In these procedures, reduction in LoS ranged between 11% and 20%. The estimation results for the control variables are mixed and do not indicate a clear pattern of association with LoS.The decrease in LoS freed resources to treat other patients, which may have resulted in reduced waiting times. It may have been more feasible to reduce LoS for urological procedures since these had relatively long LoS. Policymakers should pay attention to the effects of decreases in LoS on quality of care. Stretched hospital resources, capped reimbursements, retrospective subsidies and underpriced procedures may have limited hospitals' ability to reduce LoS for other procedures where no decrease occurred .The online version contains supplementary material available at 10.1186/s13584-022-00515-y. Means of payment affect healthcare provider decision-making, including patient admission policies and treatment decisions , 2. PaymEvidence on the effects of DRGs on healthcare providers’ behavior as measured by LoS and volume of patients has been mixed, with different results in different countries –10. In cUniversal health coverage in Israel is based on National Health Insurance (NHI), in which four competing health plans (HPs) are responsible for purchasing healthcare to all residents. Hospital revenue comes primarily from the sale of services 90% in 2017), of which 84% is paid by health plans (HPs). The met, of whicOne peculiarity of Israel’s hospital payment system is that there are two major constraints on hospital revenues. The first is an annual cap on payments by each HP. If payments exceed the cap, hospitals receive only a fraction of payment for the difference. Further constraints are imposed by alternative reimbursement contracts negotiated between HPs and hospitals , represePRG payments in Israel were defined for 30 common procedures in the 1990’s. In 2010, the MoH advocated for a broader adoption of PRGs in place of PDs. Payment reform was implemented in three waves over several years, each for different clinical domains. The first between 2010 and 2012 created about 80 new PRG codes, mostly for orthopedic urgent care procedures. The second wave was between 2013 to 2014, when the MoH created 65 PRG codes to pay for procedures mainly in urology, general surgery, ophthalmology, head and neck surgery. The last wave in 2015 covered 50 elective orthopedic procedures.The PRG-payment expansion was intended to enhance transparency by precise activity coding and improved distribution of funds using a refined unit of payment method. The intent was to reduce waiting times , to incrInitial research in Israel examined changes in patient turnover and LoS for certain procedures after PRGs were first introduced in the 1990’s  and afteWaitzberg and colleagues  examinedTo address this potential limitation, we examined the impact of PRG-based payments on average LoS at the procedure level. Based on previous findings, we expected payment change to lower costs per case by reducing LoS. We also examine change in 14 procedures less frequently examined in the literature. By expanding types of procedures, we may pinpoint how clinical fields respond differently to payment reform. And though the effects of DRG-like payments have been widely studied in other contexts with inconsistent findings across countries, Israel’s unique hospital market allows for the analysis of the potential effects of economic incentives on hospitals with capped reimbursements, stretched resources, and underpriced procedures . We take a different approach than previous research by applying survival analyses to LoS data, where the “event” is the discharge from the hospital instead of death as common in survival analysis in medical research, and the duration is the LoS instead of the survival time. In this approach, the effect of the reform is modeled as the relative change in average LoS when comparing the periods before and after the reform. We control for age, gender, comorbidity and hospital ownership and location. Technically, this approach is more suitable than for example linear models because LoS is often right-skewed and cannot be negative . In addition, the MoH was not able to distill data when the grouping of ICD-9 codes changed by PRG code.Procedures performed in inpatient settings only, as hospitals do not report ICD-9 and PRG codes for procedures performed in outpatient settings and the MoH could not distill data from this settingElective procedures only, because hospitals can choose the type of payment (PRG or PD) for urgent care. When a procedure is underpriced, and has long stays, the hospital has incentives to charge PD instead of PRG, and the effects of the reform are not seen.No trauma procedures, since multiple procedures are often performed, and hospitals only report the main procedure. In these cases, some procedures might not be properly charged, and again the economic incentives of the reform are distorted.To analyze the impacts of the second wave of the PRG reform, when many PRG codes were created in July 2013 and January 2014, we used administrative data from the MoH for selected procedures performed between 2005 and 2016 in all non-profit hospitals (97% of the acute care beds). Of the 65 PRG codes created in this period, we sampled 14 procedures , urology (7) and gynecology (2), and were selected and pooled based on the ICD9 codes listed in Table A1 in the Additional file We consider the top 5% of reported LoS as outliers and decided to drop them because they likely represent the most severe, complicated cases, where medical considerations usually takes precedence over cost. Including these cases could obscure tradeoffs between economic and medical considerations. Computing the 95% quantiles for LoS separately for each procedure yielded that LoS was up to the 26-fold of the respective 95% quantiles for some procedures. Including such severe outliers could distort the algorithm and bias estimation results even when assuming a distribution which is designed for right-skewed data.To estimate the effects of the payment reform on LoS, we performed survival analyses using a Weibull distribution, where we model the scale parameter <1). See for an exWe opted for a Monte-Carlo-Markov-Chain (MCMC) based Bayesian analysis using the Metropolis–Hastings-algorithm, because data were grouped by age and sex, and comprised summary statistics (mean LoS and median Charlson index). This means that observed variance for explanatory and outcome variables within the groups defined by age, sex, hospital and year is zero, and estimates of overall variance for explanatory and outcome variables may be attenuated. Estimation approaches in which standard errors derived from the variance of the error term would thus likely yield unreliable confidence intervals and potentially biased results.The most important difference between the frequentist and Bayesian approach used here is that the latter considers estimated parameters as random variables drawn from a distribution to be estimated. Therefore, our estimation aimed to describe the distributions of parameters reflecting observed patterns in the data. These distributions are used to derive expected parameter values and the probability that a parameter lies within a certain interval. In the frequentist approach, uncertainty is mainly the result of sample variations, where the variance of a parameter is the outcome of randomness in the sampling process, such that different samples yield different results, but the perfect (infinite) sample would find the true parameter with certainty. The frequentist approach thus considers parameters as fixed but unknown, and assumes that confidence intervals include the fixed but unknown parameter with a certain probability.As priors, we use fairly, but not fully, uninformative distributions. We drew fixed effect parameters in the MCMC simulation based on a normal distribution with mean = 0 and standard deviation Table The descriptive statistics in table two indicate that data before and after the 2013–2014 reform on patient demographics did not change noteworthily. The male/female-ratio was similar before and after the reform. The median age increased for four procedures, but not in the overall sample. The median Charlson index changed only for open splenectomy. In other words, any change in LoS is unlikely to be the result of changes in the demographic profile of the patients over time.Table Table Since interpretation of parameter estimates for Bayesian analyses differs from frequentist analyses, we also employed a different approach to hypothesis testing. Analogously to parametric hypothesis-testing, we applied both 95% and 99% certainty thresholds to identify time ratios significantly < 1 . The third column in Table a-posteriori cumulative distribution functions (CDF) of the estimated time ratios. For example, the CDF in the plot for open splenectomy is 96.8% for a time ratio of 1, indicating that the probability for decreased LoS is 96.8%. For most procedures, the curves exhibit the steepest increase for time ratios between 0.8 and 1, suggesting high probabilities for a shortened length of stay even for some procedures where changes were not significant. The intersections of the curves with the dashed lines correspond to the 95% credibility intervals reported in Table Figure Tables A4–A6 in the Additional file The shape parameter To assess the reliability of results, we examined the simulated MCMCs to see whether they converged and mixed .a-posteriori distributions. In addition, the acceptance rates of all procedures ranged from 0.3 to 0.4, which indicates a good performance of the Metropolis–Hastings-algorithm. Higher values would indicate too small steps in the search algorithm and bear the risk of omitting part of the potential parameter space, whereas too low acceptance rates would point towards too large steps in the search algorithm, which would indicate a random search pattern and the risk of, say, rushing too fast through the potential parameter space.First, we used effective sample sizes reported in Table To assure convergence, we also examined various plots to graphically assess whether MCMCs were well behaved and whether convergence could be observed. First, MCMCs for all estimated parameters across all procedures yielded no visible trends or heteroscedasticities, suggesting converged (stationary) processes. Second, histograms of MCMC samples yielded unimodal parameter distributions, which again suggests a converged MCMC, and would be unlikely in cases of non-convergence. Third, the number of lags in autocorrelation functions suggested only a few statistically significant lags in MCMCs, again implying that they converged during the burn-in. Finally, dividing MCMCs in half and comparing kernel densities for simulated values yielded extremely similar patterns, which again indicates that mean, standard deviation and the shapes of the distributions did not significantly differ across the simulation process. This further supports the assertion that MCMCs converged during the burn-in process and followed a stationary process throughout the MCMC simulation process.We estimated the impact of change in hospital payment from PDs to PRGs on LoS. We analyzed 14 general surgery, urology and gynecology procedures with mixed findings. Introduction of PRGs was intended, inter alia, to create incentives to reduce costs per case and thereby reduce LoS. Hospitals reduced LoS for half for procedures examined, mostly urological procedures. For those procedures, LoS declined by 11% to 20%. One potential explanation is that most urology procedures analyzed are more invasive; i.e., open procedures, which are characterized by long LoS, mainly due to lengthy recovery. In contrast, most of the general surgical procedures we examined were laparoscopic, and characterized by shorter LoS and faster recovery. It is more feasible to reduce LoS for invasive procedures by focusing on recovery, than laparoscopic procedures where the LoS are already short –34.th and 99th percentile exhibit only minor changes. This supports our argument that the reform had the strongest impact on “average patients”, while decisions concerning more complicated or more severe cases may have been less subject to economic considerations.Inspecting the entire distributions of the length of stay before and after reform indicates that the reform had its main effect in the middle of the distributions around the mean and median of LoS, whereas the 95Similar to European countries, one objective of introducing PRGs in Israel was to reduce waiting times . By settIt is possible that the decrease in LoS released some resources to treat other patients, maybe reducing waiting times. Since the number of beds in Israel did not decrease , it is pFor this study, we performed survival analyses to estimate relative changes in LoS. This methodology is particularly well-suited to these data and research questions. First, LoS are time-until-event data, for which survival analysis and underlying distribution assumptions were specifically designed. Second, measuring relative change in LoS in a context of payment reform allowed us to examine various patterns of LoS across procedures and patient groups. Patients with more comorbid conditions, for example, were likely to have longer LoS both before and after reform. We argue that similar relative changes by age, sex and comorbidity may be more likely than similar absolute changes, although we acknowledge that mixed effects linear regression models, for example, might yield comparable results. Third, since our observations were grouped by age, gender, procedure, hospital and year , no variance was observed within those groups. A frequentist approach would underestimate the error variance and lead to biased results. The Bayesian approach allowed us to overcome the problem of lack of variance within the data groups because it is a simulation-based approach which compares many simulated outcomes with the observed real-world data, and does not require the computation of error covariance matrices.x-axis and observe the resulting probabilities on the y-axis in Fig. Finally, we considered the Bayesian estimation to provide more relevant results for policymakers, since we present credibility intervals and cumulative distribution functions in addition to the usually reported point estimates for each procedure. In particular, Fig. Our findings complement existing PRG reform research in Israel. Using a diff-in-diff analysis at the ward level, Waitzberg et al.  found noIn a qualitative study examining the effects of PRG reform, hospital managers and physicians reported that reimbursement for many PRGs were insufficient to cover costs, which tempered incentive for hospitals to increase patient turnover. The exception they reported was for orthopedic and urological procedures, for which LoS decreased and patient turnover increased, as found here. PRG-based payments alone do not incentivize patient turnover if prices do not allow for some marginal profit . In factIt is widely acknowledged that shortening LoS through economic incentives may negatively affect the quality of care, especially in cases where patients are released too early. Israeli policymakers should therefore pay attention to the effects of decreases in LoS and assure that these decreases do not result in poorer health outcomes. Unfortunately, the MoH monitors indicators of process and quality of care at the hospital level, and only for some procedures, which do not include those analyzed in this study, 40. If The finding that LoS did not significantly decline for half of the procedures examined  can be explained by various moderating factors which attenuated the effects of the PRG reform . The firThese findings are applicable to other systems since these same moderating factors may impede payment reform to achieve desired objectives in other countries, or lead to unintended consequences. Nation-specific factors from other countries should be considered when implementing DRG-like payment programs.Our findings are in line with previous research suggesting that DRG-like payment reform has limited and inconsistent effects on hospital activity; results vary depending on context and country . For exaDRG payments have different effects in different countries and contexts. Simplistic evaluations of DRGs and facile comparisons between countries can lead to biased interpretations, inappropriate and adverse policy and political consequences . New proFirst, we did not have access to all costing data for procedures. Data on the actual costs of hospital care at the national level in Israel and other countries is limited, if they exist at all. Acquiring cost data from each hospital was beyond the scope of this study. This lack has two implications. First, the differences between pricing (payment tariff) and cost might be small, non-existent or negative, thus undermining incentives for change in admission and treatment policy. Therefore, our study cannot pinpoint the reason for the (lack of) changes in LoS. Second, we were unable to observe the exact reimbursements and the corresponding annual caps for the hospitals. We include the year-fixed effects to capture potential year-specific effects, and argue that annual changes in maximum reimbursements are adequately addressed by this methodology.Second, our data were grouped by procedure and age-and-sex groups at the hospital level, not the patient-level; we did not have individual LoS data and could not assess LoS variance within groups. However, with the data grouped by age, sex, procedure, hospital and year, the uninflated dataset (i.e. before accounting for group sizes) already comprised 24,287 data points, which corresponded to 87,559 individual cases. There is a tradeoff between detailed, patient-level data from a few hospitals which might not be representative versus less detailed data from all hospitals. We opted for the latter to assess the impact of payment reform at the national level, and attempted to overcome this data limitation by Bayesian estimation.Third, the data structure and the extreme outliers  led to problems with the estimation algorithms. The raw data indicates that the maximum stays and the 95% quantiles decreased to some extent after reform, but the main effect of the reform was observed around the mean and median of the distributions. Outliers were computed for each procedure over the full period. The fraction of observations considered as outliers varies around 5% over the years and shows no clear shift around the year of reform.Fourth, our analyses were restricted to 14 procedures and inclusion was limited to procedures with 200+ cases per year to ensure that findings would not merely be a result of stochastic processes. Changes in LoS may have occurred for procedures not included in our analyses, in particular those for which reimbursements are still made per diem. While the 14 procedures might not provide a full picture of hospitals’ activities, by analyzing only procedures created in the second wave, the simultaneous introduction of the PRG payment allows a more reliable comparison between the various procedures.Fifth, we excluded private hospitals, which represent 3% of the beds, but about 30% of the procedures . These hFinally, the data included no information about the supplier characteristics. Reliable information about the numbers of surgeons, the number of hospital beds per ward or the actual waiting times were not available. We are not aware of noteworthy changes in the hospital infrastructure of or noteworthy changes in the demographic characteristics of the population, however, and argue that including years as random effects part in the fixed effects model would have covered such unobserved, year-specific information.Analyses of the effects of DRG-like payments can be more complex than first assumed. The effects of different payment mechanism may differ across different procedures. This depends on the grouping of patients, pre-reform LoS, the pricing, the technology (invasive or laparoscopic), demand elasticity and many other non-economic considerations such as physicians’ preferences, and commitment to the patient, to mention only a few. We found significant decreases in LoS for invasive procedures with longer recovery times whereas laparoscopic procedures with already short LoS showed no significant change; LoS cannot drop below a certain minimum. Conclusions as to payment reforms should thus be made with caution and consideration for varied effects across procedures, wards, and hospitals. Finally, context matters and country-specific factors can promote, moderate or impede responses to economic incentives. Aggregate and over-simplified data can bias findings and mislead policy makers.Additional file 1. Appendix.


Pooling cells from multiple biological samples prior to library preparation within the same single-cell RNA sequencing experiment provides several advantages, including lower library preparation costs and reduced unwanted technological variation, such as batch effects. Computational demultiplexing tools based on natural genetic variation between individuals provide a simple approach to demultiplex samples, which does not require complex additional experimental procedures. However, to our knowledge these tools have not been evaluated in cancer, where somatic variants, which could differ between cells from the same sample, may obscure the signal in natural genetic variation.in silico benchmark evaluations by combining raw sequencing reads from multiple single-cell samples in high-grade serous ovarian cancer, which has a high copy number burden, and lung adenocarcinoma, which has a high tumor mutational burden. Our results confirm that genetic demultiplexing tools can be effectively deployed on cancer tissue using a pooled experimental design, although high proportions of ambient RNA from cell debris reduce performance.Here, we performed in silico benchmark evaluations, available at https://github.com/lmweber/snp-dmx-cancer.This strategy provides significant cost savings through pooled library preparation. To facilitate similar analyses at the experimental design phase, we provide freely accessible code and a reproducible Snakemake workflow built around the best-performing tools found in our  Sample pooling prior to library preparation is an effective strategy for experimental design in single-cell RNA sequencing (scRNA-seq) studies, which allows researchers to assess and address unwanted technological variation such as batch effects , 2 and rExisting demultiplexing approaches differ in their experimental procedures, computational methodology for demultiplexing, and demultiplexing accuracy. In barcoding-based approaches  and 5 ss  and lung adenocarcinoma (high SNV) using a set of benchmark evaluations and Snakemake  workflowUsing the HGSOC scRNA-seq and matched bulk RNA-seq data, we found the highest recall (defined as the proportion of true singlet cells for each sample that are identified as singlets and assigned to the correct sample) and best precision-recall trade-off (where precision is defined as the proportion of identified cells for each sample that are true singlet cells from the correct sample) when using bcftools  to generIn general, we prefer higher recall at the expense of somewhat lower precision, so that we do not lose true singlet cells during the initial demultiplexing. If reduced precision is due to additional doublets that have been misclassified as singlets, these can potentially be identified and removed through downstream analyses, such as inspecting visualizations of unique molecular identifier (UMI) counts or detected genes per cell, or applying downstream doublet detection tools . As an iIn the comparisons with demuxlet, we found that using bcftools  to generIn the scenarios where matched bulk RNA-seq samples are not available, the next best-performing scenarios were obtained using the genotype reference from the 1000 Genomes Project  provide–C. The “Using the high-TMB lung adenocarcinoma scRNA-seq (without matched bulk RNA-seq) dataset, we only considered the scenario using the genotype reference from the 1000 Genomes Project (filtered) together with cellSNP/Vireo  consortium and calculated the overlapping sets of SNPs from this array with either the 1000 Genomes filtered (“1000GenomesFiltMEGA”), 1000 Genomes unfiltered (“1000GenomesUnfiltMEGA”), or bulk RNA-seq (“bulkBcftoolsMEGA”) reference (for the HGSOC dataset). This left either 16.5% of the 84,853 filtered 1000 Genomes SNPs, 8.6% of the 7,414,539 unfiltered 1000 Genomes SNPs, or 7.6% of the 605,367 bulk RNA-seq SNPs, or alternatively (compared to the array) 0.8%, 36.8%, or 2.6% of the original 1,733,345 array SNPs, respectively see . BecauseAs a baseline comparison for healthy (non-cancer) data, we evaluated performance in a dataset consisting of 5 samples of induced pluripotent stem cell (iPSC) cell lines from the Human iPSC Initiative (HipSci), which was previously published by Heaton et al. . This daWe evaluated the computational runtimes for the various components in our benchmark scenarios and Snakemake workflow using the HGSOC data. First, we found that the computational runtimes for the various steps in the genetic demultiplexing workflow vary across multiple orders of magnitude and depended on whether the tool could be parallelized. The parallelizable tools (Cell Ranger and cellSNP) were run using 10 processor cores to decrease runtime, while the remaining tools used a single core. All evaluations of runtimes were performed on a high-performance Linux computing cluster. In the Snakemake workflow Fig. , the sloWe also evaluated computational runtimes for the genotyping tools used to generate the genotype reference lists of SNPs from either the matched bulk RNA-seq samples or directly from the scRNA-seq samples Fig. . Here, w–based computational sample demultiplexing are a convenient and effective strategy for reducing library preparation costs and potential batch effects in scRNA-seq studies. Compared to barcoding-based approaches such as MULTI-seq [–based methods provide significant advantages in terms of simpler sample preparation and cost savings during library preparation. Here, we performed an in silico benchmark evaluation based on real scRNA-seq tumor tissue datasets to confirm that these tools can be effectively applied to pooled cancer samples from different individuals. We compared 2 demultiplexing tools  and 5 gs  when using matched bulk RNA-seq samples to generate a genotype reference list of SNPs using bcftools , togethein silico evaluations, although in this case it may be difficult to generate a reliable ground truth.Our study has several limitations. While the best-performing benchmark scenario achieves excellent recall, precision is somewhat lower. While most doublet calls from cellSNP/Vireo in this scenario were true identifiable doublets, additional true identifiable doublets were incorrectly called as singlets, reducing precision for each demultiplexed sample. Although downstream doublet detection tools  could beMore fundamentally, due to the reliance on genetically distinct SNP profiles, genetic demultiplexing tools work well for human samples from unrelated individuals, but performance is expected to decrease for genetically similar samples such as hereditary related human populations  or inbreWe begin by describing in detail our benchmark evaluation framework, and we note that our additional Snakemake  workflowIn our benchmark evaluation, we considered 2 genetic demultiplexing algorithms: (i) Vireo  togetherFor the main benchmark evaluations, we used 2 cancer datasets. The first dataset consists of 3 UMI-based scRNA-seq HGSOC samples measured on the 10x Genomics platform , obtainePerformance was evaluated in terms of precision and recall for demultiplexing each scRNA-seq sample. We also recorded computational runtime for each step in the workflow and benchmark scenarios. Recall is defined as the proportion of true singlet cells for each sample that are identified as singlets and assigned to the correct sample. Precision is defined as the proportion of identified cells for each sample that are true singlet cells from the correct sample. Runtime was evaluated using the Unix “date" command. We used R version 4.1 for random number generation and evaluation steps performed in R, and created figures using ggplot2 .in silico simulation scenarios for each dataset—containing either no doublets, 20% simulated doublets, or 30% simulated doublets. Doublets were simulated by combining cell barcode labels from random sets of 2 cells in the raw sequencing reads mapped using Cell Ranger [For our benchmark evaluation, we developed 3 l Ranger , so thatl Ranger , 17, 29;For the scenarios containing ambient RNA from simulated cell debris or lysed cells, we selected a percentage of cell barcodes  after doublet creation and assigned all sequencing reads from these cell barcodes to other randomly selected cell barcodes, in each of the no doublets, 20% doublets, and 30% doublets scenarios. The debris percentages  were selected to be in the higher range of previously published results for non-cancer data , since aFor the simulated SNP array analyses, we selected a widely used SNP array  and calculated the overlapping sets of SNPs between the total 1.7 million SNPs from the array and our existing genotype references  data, we combined sequencing reads from 5 samples of induced pluripotent stem cell (iPSC) cell lines from the Human iPSC Initiative (HipSci), which were previously published by Heaton et al. , and addDe-identified HGSOC samples were processed after cryopreservation in liquid nitrogen where tissue chunks were stored in RPMI medium with 10% fetal bovine serum and 10% DMSO. Samples were thawed and dissociated to single cells using the Miltenyi Human Tumor Dissociation Kit and the GentleMACS dissociator. Samples were incubated on the GentleMACS at 37°C for 1 hour with the setting of 1,865 rounds per run. A 70-µm MACS smart strainer was used to deplete cell doublets before loading onto the 10x Genomics Chromium Controller. Library preparation was performed using the 10x Genomics 3′ Gene Expression Library Prep v3 and libraries were sequenced on an Illumina NovaSeq instrument.Project name: snp-dmx-cancerhttps://github.com/lmweber/snp-dmx-cancerProject home page: Operating system: LinuxProgramming language: Shell, ROther requirements: High-performance computing cluster with Sun Grid Engine schedulerLicense: MITGigaScience database GigaDB [All code and scripts to reproduce the benchmark evaluations, supplementary analyses, generate figures in the manuscript, and run the Snakemake workflow are freely accessible from GitHub . An arche GigaDB . All tooRaw and processed sequencing data generated in this study (HGSOC dataset) are available from the Database of Genotypes and Phenotypes (dbGaP)  and Gene Expression Omnibus (GEO) . The lung adenocarcinoma dataset was previously published by Kim et al.  and is aSupplementary Figure S1. Performance evaluations for benchmark scenarios including ambient RNA from simulated cell debris. Top-performing and computationally efficient scenarios for HGSOC dataset (A–C) and lung adenocarcinoma dataset (D–F), across 3 proportions of simulated doublets , after introducing ambient RNA from simulated cell debris by assigning all reads from 20% of final cell barcodes to other randomly selected cell barcodes. Performance is evaluated in terms of precision (y-axis) and recall (x-axis) for recovering the sample identities of true singlet cells from each scRNA-seq sample. Benchmark scenarios are labeled by color and with the naming scheme “genotypeMethod_demultiplexingMethod.” Samples within each dataset are identified with shapes. Axis limits range from 0 to 1 for all panels.Supplementary Figure S2. Performance evaluations for benchmark scenarios including ambient RNA from simulated cell debris. Top-performing and computationally efficient scenarios for HGSOC dataset (A–C) and lung adenocarcinoma dataset (D–F), across 3 proportions of simulated doublets , after introducing ambient RNA from simulated cell debris by assigning all reads from 40% of final cell barcodes to other randomly selected cell barcodes. Note that unlike Fig. Supplementary Figure S3. Performance evaluations for benchmark scenarios using subset of SNPs from SNP array as genotype reference. Top-performing and computationally efficient scenarios for HGSOC dataset (A–C) and lung adenocarcinoma dataset (D–F), across 3 proportions of simulated doublets , when using a subset of SNPs from an SNP array  overlapping with either the 1000 Genomes Project filtered (“1000GenomesFiltMEGA”), 1000 Genomes Project unfiltered (“1000GenomesUnfiltMEGA”), or bulk RNA-seq (“bulkBcftoolsMEGA”) reference as the genotype reference for demultiplexing. Performance is evaluated in terms of precision (y-axis) and recall (x-axis) for recovering the sample identities of true singlet cells from each scRNA-seq sample. Benchmark scenarios are labeled by color and with the naming scheme “genotypeMethod_demultiplexingMethod.” Samples within each dataset are identified with shapes. Axis limits range from 0 to 1 for all panels.Supplementary Figure S4. Performance evaluations for healthy (non-cancer) cell line dataset. Top-performing and computationally efficient scenario from main results (“1000GenomesFilt_cellSNPVireo”) for healthy (non-cancer) induced pluripotent stem cell (iPSC) cell line dataset sourced from [ced from , consistSupplementary Figure S5. Illustration of expected cost savings from multiplexed experimental design prior to library preparation. The figure shows the total of estimated library preparation and sequencing costs, with either no multiplexing or full multiplexing , for experiments with 4, 6, or 8 samples. The calculations assume 4,000 desired cells per sample after demultiplexing, after discarding identifiable doublets consisting of cells from multiple samples; library preparation costs of $2,000 per sample or multiplexed set of samples; sequencing costs of $1,500 per 400 million reads with an additional 30% cost due to unaligned reads and adapters; and ∼20,000 reads per cell. Calculations were performed using the “Cost Per Cell” online calculator provided by the Satija Lab [tija Lab .Supplementary Table S1. Number and percentage of SNPs in sets overlapping between the available genotype references . The bulk RNA-seq and 1000 Genomes filtered references are used for the main results.Supplementary Table S2. Summary of HGSOC, lung adenocarcinoma [arcinoma , and heaarcinoma  datasetsSupplementary Table S3. Confusion matrix for singlet, doublet, and unassigned calls for the top-performing scenario (cellSNP/Vireo with bulk RNA-seq genotype reference) for 30% doublets scenario for HGSOC dataset  for 30% doublets scenario for HGSOC dataset  to demultiplexed cells after applying top-performing and computationally efficient demultiplexing tools , for 20% and 30% doublets scenarios. scDblFinder was run using default settings, and clusters representing doublets identified using thresholds of 70 (HGSOC) and 10 (lung adenocarcinoma) differentially expressed genes per cluster based on inspection of elbow plots. False-positive and false-negative doublet calls are shown in bold font.CNV: copy number variation; DMSO: dimethyl sulfoxide; HGSOC: high-grade serous ovarian cancer; iPSC: induced pluripotent stem cell; MEGA: Multi-Ethnic Genotyping Array; RPMI: Roswell Park Memorial Institute; scRNA-seq: single-cell RNA sequencing; SNP: single-nucleotide polymorphism; SNV: single-nucleotide variant; TMB: tumor mutational burden; UMI: unique molecular identifier.Ovarian cancer tissue was obtained and studied under written informed consent at the Huntsman Cancer Institute through approved University of Utah Institutional Review Board protocols IRB_00010924 and IRB_00118086. Analysis of human data in this study was also approved by the University of Pennsylvania Institutional Review Board (IRB protocol 832353) and the Johns Hopkins Bloomberg School of Public Health Institutional Review Board (IRB00013099).The authors declare that they have no Conflict of Interest.L.M.W., A.A.H., K.C.B., J.G., J.A.D., C.S.G., and S.C.H. were supported by the National Institutes of Health grant from the National Cancer Institute R01CA237170. J.A.D. is also supported by Huntsman Cancer Foundation and National Institutes of Health grant from the National Cancer Institute P30 CA042014 (to N. Ulrich).L.M.W.: Software, Formal analysis, Investigation, Data curation, Writing—Original Draft, VisualizationA.A.H.: SoftwareP.F.H.: Software, Writing—Review & EditingK.C.B.: InvestigationJ.G.: Investigation, Resources, Writing—Review & EditingJ.A.D.: Resources, Writing—Review & Editing, Funding acquisitionC.S.G.: Conceptualization, Resources, Writing—Review & Editing, Funding acquisitionS.C.H.: Conceptualization, Resources, Writing—Original Draft, Writing—Review & Editing, Supervision, Funding acquisitiongiab062_GIGA-D-21-00074_Original_SubmissionClick here for additional data file.giab062_GIGA-D-21-00074_Revision_1Click here for additional data file.giab062_Response_to_Reviewer_Comments_Original_SubmissionClick here for additional data file.giab062_Reviewer_1_Report_Original_SubmissionJan-Philipp Mallm -- 3/30/2021 ReviewedClick here for additional data file.giab062_Reviewer_1_Report_Revision_1Jan-Philipp Mallm -- 7/30/2021 ReviewedClick here for additional data file.giab062_Reviewer_2_Report_Original_SubmissionHaynes Heaton, MD -- 4/2/2021 ReviewedClick here for additional data file.giab062_Reviewer_2_Report_Revision_1Haynes Heaton, MD -- 8/5/2021 ReviewedClick here for additional data file.giab062_Supplemental_FileClick here for additional data file.


In high-dimensional survival genomic data, identifying cancer-related genes is a challenging and important subject in the field of bioinformatics. In recent years, many feature screening approaches for survival outcomes with high-dimensional survival genomic data have been developed; however, few studies have systematically compared these methods. The primary purpose of this article is to conduct a series of simulation studies for systematic comparison; the second purpose of this article is to use these feature screening methods to further establish a more accurate prediction model for patient survival based on the survival genomic datasets of The Cancer Genome Atlas (TCGA).Simulation studies prove that network-adjusted feature screening measurement performs well and outperforms existing popular univariate independent feature screening methods. In the application of real data, we show that the proposed network-adjusted feature screening approach leads to more accurate survival prediction than alternative methods that do not account for gene-gene dependency information. We also use TCGA clinical survival genetic data to identify biomarkers associated with clinical survival outcomes in patients with various cancers including esophageal, pancreatic, head and neck squamous cell, lung, and breast invasive carcinomas.These applications reveal advantages of the new proposed network-adjusted feature selection method over alternative methods that do not consider gene-gene dependency information. We also identify cancer-related genes that are almost detected in the literature. As a result, the network-based screening method is reliable and credible. In high-dimensional genomic data, identifying important genes related to clinical survival traits is a challenging and important problem in the field of bioinformatics. The discovery of important biomarkers that explain the phenotype of interest is essential for the development of phenotype prediction models. In particular, contaminated data and right-censored survival outcomes make relevant feature screening more challenging.In the era of high-throughput biology, the number of potential features/biomarkers could be much larger than the research sample size. In this case, it is well-known that use of preliminary feature screening can substantially improve the model selection performed by the regularization approach . UnivariHowever, outlier-contaminated biomarker data pose a further challenge to the survival prediction problem based on high-dimensional genetic/genomic data. As is known, the developmental process of disease is complicated and may involve the interaction of multiple genes; that is, epistasis. In other words, to effectively identify disease-related genes, it is necessary to make full use of biological network information. In this article, we intend to perform a systematic comparison for these advanced feature screening methods and apply them to The Cancer Genome Atlas TCGA,  survivalWe consider a study with MVS” that can be downloaded from https://github.com/thomashielscher/MVS, which contains the first six screening methods discussed in https://figshare.com/articles/software/CODE/16677070, laying the foundation for meaningful comparison.We list the common and effective survival feature screening methods, and provide readers with an overview summarized in https://figshare.com/articles/dataset/DATA/16677619. The TCGA HNSCC genomic data can be downloaded from the R package “GEInter”  statistics among 200 replications through violin plot to view the distribution of MMS data and its summary statistics, where MMS is the minimum size of a selected model, including underlying effective predictors. MMS measures the complexity of the selected model and reflects the accuracy of the screening process; a smaller MMS value indicates the higher accuracy of feature screening. We note that a violin chart can be constructed through the “ package . We also = |A∩B|/|A∪B|) of the true feature list is calculated by selection of by two methods under a specific model size of 500. The average Jaccard index of 200 simulation repetitions is used as a measure of similarity between the two methods; in this way, the similarity matrix of all the screening methods under consideration can be constructed and visualized, for example, using the “corrplot” R package  = |A∩B|/min ) set similarity analysis of features selected by each method with a ground truth set of predictors. The average number of overlap coefficient index among 200 replications as a function of the number of selected features for each method is used as a measure of similarity between the feature screening method and the ground truth set of predictors. A larger overlap coefficient index indicates highly similarity with a ground truth set of predictors.In addition, we calculate the overlap coefficient  metrics. Larger c-index/smaller deviance and number of selected features indicates better prediction accuracy. Since the c-index metric can be used to compare a subject’s predicted survival time ranking with the true survival time ranking. And, the deviance metric is defined asTo evaluate the performance of survival prediction, we report three measures of prediction accuracy: the The deviance metric can therefore be considered as a comparison between the survival prediction model and null model (no predictors considered). The deviance metric is also a suitable survival prediction criterion.Finally, we choose the number of selected features metric as a precision criterion; the reason is that the feature screening approach is used to find parsimonious precision models. Meaning, if we are modeling two sets of features with the same predictive accuracy, we want to choose the model that uses the features, as a smaller number of selected features are easier to interpret/evaluate in follow-up studies about biological function.In simulation studies, a series of simulation studies are conducted to investigate the performance of the existing feature screening methods for survival trait in identifying true associated predictors and survival prediction errors.The simulation results are summarized in c-index when the number of selected variables is larger, and the IPCW-tau (NPN-MB) method outperforms most alternative methods when the number of selected variables is medium or small.From simulation studies 1, 2, and 3 with band structure, we see that the IPCW-tau (NPN-MB) method outperforms all alternative methods in terms of the overlap coefficient  method performs the best when the number of selected variables is larger or medium in terms of the overlap coefficient  and all other settings are the same as those in the previous simulation study. We still obtain similar numeric results and the proposed IPCW-tau (NPN-MB) method has a better performance than the alternative methods at most gene structures. These correspondence figures are all available at Figshare: https://figshare.com/articles/figure/Survival_Feature_Screening_based_on_proportional_odds_model/17089013.A further simulation study is conducted under the scenario where the survival time follows a proportional odds (PO) model  related to ESCA  genes that are related to ESCA in the literature  method for whole data to identify several important biomarker genes and estimate the correspondence parameters by penalized Cox’s regression model with the MCP penalty. Please see terature .p-values based on marginal Cox’s model are selected for downstream analysis. We take five random splits of the whole data into 142:36 training/test sets of the data to evaluate the performance of all methods for survival prediction in the TCGA PAAD data.After excluding patients with missing survival time data, our analysis is focused on the subset of the TCGA PAAD data with 178 patients and 20,501 gene expression variables. The censoring rate in the data is about 48%. As the number of disease-associated biomarkers is not expected to be large, the top 2,000 genes with the smallest CDKN2A, TP53, TTN, KRAS) related to PAAD  and find the MET gene that is related to PAAD in the literature  method for whole data to identify several important biomarker genes and estimate the correspondence parameters by penalized Cox’s regression model with the MCP penalty. Please see terature .GIMAP6, SELL, TIFAB, KCNA3, CCR4) related to HNSCC  related to LUAD  related to BRCA  that are related to HNSCC in the literature  that are related to LUAD in the literature  that are related to BRCA in the literature  method outperforms most alternative methods when the number of selected variables is medium or small. Although in the real data application, the c-index measure of the IPCW-tau (NPN-MB) method is not always the best among all considered feature screening methods or the published biomarker gene models, the IPCW-tau (NPN-MB) method still outperforms most alternative methods by the different prediction metrics. In addition, we observe every method can also identify the biomarker genes that are related to TCGA cancer in the literature. However, according to the simulation studies and real data analysis, we can still infer that the IPCW-tau (NPN-MB) method has better performance in both variable selection and survival prediction. To this end, the IPCW-tau (NPN-MB) method is a good choice for developing a survival prediction model.According to simulation studies for the c-index for prediction errors, overlap coefficient index, and the Jaccard index. Moreover, in real data applications, The main purpose of p-value for downstream analysis based on the marginal Cox’s model. Different candidate models lead to different survival prediction models. How to define the number of cancer-related biomarkers for downstream analysis is a key and interesting issue.In the real data application, we adopt the hard thresholding rule proposed by There are several public human databases, like METABRIC, TCGA, NCDB, and GEO. These useful databases can be utilized to determine the reproducibility of our findings. We consider that meta-analysis can be performed for discovery and validation of survival biomarker genes , which i10.7717/peerj.13098/supp-1Supplemental Information 1Click here for additional data file.


A major challenge in drug development is safety and toxicity concerns due to drug side effects. One such side effect, drug-induced liver injury (DILI), is considered a primary factor in regulatory clearance. The Critical Assessment of Massive Data Analysis (CAMDA) 2020 CMap Drug Safety Challenge goal was to develop prediction models based on gene perturbation of six preselected cell-lines (CMap L1000), extended structural information (MOLD2), toxicity data (TOX21), and FDA reporting of adverse events (FAERS). Four types of DILI classes were targeted, including two clinically relevant scores and two control classifications, designed by the CAMDA organizers. The L1000 gene expression data had variable drug coverage across cell lines with only 247 out of 617 drugs in the study measured in all six cell types. We addressed this coverage issue by using Kru-Bor ranked merging to generate a singular drug expression signature across all six cell lines. These merged signatures were then narrowed down to the top and bottom 100, 250, 500, or 1,000 genes most perturbed by drug treatment. These signatures were subject to feature selection using Fisher’s exact test to identify genes predictive of DILI status. Models based solely on expression signatures had varying results for clinical DILI subtypes with an accuracy ranging from 0.49 to 0.67 and Matthews Correlation Coefficient (MCC) values ranging from -0.03 to 0.1. Models built using FAERS, MOLD2, and TOX21 also had similar results in predicting clinical DILI scores with accuracy ranging from 0.56 to 0.67 with MCC scores ranging from 0.12 to 0.36. To incorporate these various data types with expression-based models, we utilized soft, hard, and weighted ensemble voting methods using the top three performing models for each DILI classification. These voting models achieved a balanced accuracy up to 0.54 and 0.60 for the clinically relevant DILI subtypes. Overall, from our experiment, traditional machine learning approaches may not be optimal as a classification method for the current data. Adverse drug reactions (ADRs) are a common concern of novel drugs and therapeutics. One of the more common targets of ADRs is the liver due to its role in the metabolism of compounds and resulting liver damage is termed as Drug-Induced Liver Injury (DILI) . DILI isPredictive markers of DILI, determined by compound properties and known variables rather than preclinical studies, would facilitate drug development in a wide variety of ways . MultiplBuilding upon these previous efforts to accurately predict DILI, the Critical Assessment of Massive Data Analysis (CAMDA) in collaboration with the Intelligent Systems for Molecular Biology (ISMB) has proposed the CMap Drug Safety Challenge for their annual conferences in 2018, 2019, and 2020 . The previa a super learner method  values of these models, the top three algorithms for each datatype  for each DILI class were included in our ensemble voting model. We tested hard, soft, and weighted voting across these datasets to see if the varying dimensions of data can improve predictive performance.We constructed models to predict each drug’s DILI class (positive or negative) within the four DILI classifications  by first evaluating the performance of each dataset in predicting DILI and also by employing ensemble voting with the top three performing models across data types. The gene expression data presented a unique challenge in that not all drugs were tested in each cell line or even in liver-relevant cell lines. To address this, we utilized a Kru-Bor merging method to merge the expression signatures across cell lines into one representative drug signature . These eThe overall workflow of our study is shown in The CAMDA organizers provided us with FAERS data for all 617 drug compounds. Of these, 422 were grouped as “training data”. This dataset contains 20 features corresponding to information on the percentage of reported adverse events for each drug compound by gender and age group demographic. After removing highly correlated features, we upsampled the data to cater to the class imbalance by randomly sampling with replacement from the minority class to balance the majority class. An additional preprocessing step was to create two new variables, namely “male ratio” and “female ratio”, taking into account all reported events irrespective of the gender, all reported DILI events irrespective of gender, and the percentage of reported DILI events by gender.In addition to the FAERS dataset, we were provided with concentration-response information of 600 drugs. Of these, 412 were designated as “training data”. Thirty-two features corresponded to concentration-response curve ranks. Out of all 412 drugs for training, 57 drugs were removed for missing values. In addition, we removed highly correlated features using an arbitrary cutoff of 0.82 and catered to the class imbalance by using SMOTE.Alongside the FAERS and TOX21 data provided, we had access to the 2D molecular descriptors or structural information of these 617 drug compounds. 422 of these drugs were designated for training. There were 777 features for each drug compound with each feature corresponding to MOLD2 descriptors. To cater to class imbalance, we upsampled minority classes, as well as ROSE, and SMOTE.The L1000 assay data used in this study is a high-throughput gene expression assay that measures mRNA transcript abundance of 978 landmark genes based on an inference algorithm to infer the expression of 11,450 additional genes in the transcriptome . UtiliziSince not all drugs were tested in each cell line data made available, we utilized the Kruskal-Borda (Kru-Bor) merging algorithm in the GeneExpressionSignature R package . This apvia our Kru-Bor merging was based on a gene’s significance  in predicting the DILI score via a Fisher’s exact test. If a gene is included in the top or bottom 100, 250, 500, or 1,000 ranked list, depending on the model data, for any drug it would be assigned a one (True), or if it fell outside of that range it would be assigned a zero . The classifier for each type of DILI was also one (DILI positive) or zero (DILI negative). We used these classifiers to identify if these highly influenced genes were predictive of a drug being DILI positive or DILI negative with a p-value cutoff of 0.01.A method of feature selection utilized across the merged signatures produced The prediction of DILI was treated as a binary classification problem for each DILI type. That is, for each of DILI1, DILI3, DILI5, and DILI6, outcomes were split between “positive” and “negative”. We used a 5-folds cross validation repeated 100 times, and a random search strategy to search for the best parameters for each model. The data was made available such that training and test sets had been pre-identified. Importantly, we did not have access to the correct labels for the test data. Models were built using traditional machine learning algorithms within the caret  package The machine learning algorithms we used are suitable for classification tasks. They include a Logistic Regression (LR) , Linear To evaluate the performance of our models, we focused on the area under the ROC curve (AUC) value as well as the specificity, sensitivity, accuracy, and MCC of the models on the test set. The AUC value is a widely-used metric in binary classification problems. An AUC value of one indicates a perfect classifier, i.e. a model that is perfectly able to separate both classes, while an AUC value of 0.5 indicates a model that predicts at random. Depending on the application domain, AUC values of 0.7 and above are usually acceptable. Specificity measures the ratio of negative classes that were correctly identified by the model out of all negative classes, while sensitivity measures the ratio of positive classes that were correctly identified by the model out of all positive classes. These metrics are affected by how the target labels are structured and passed to the algorithm, and they range from 0 to 1. Additionally, we evaluated the performance of our models on the test set by calculating the balanced accuracy of prediction. Balanced accuracy is the average of the sensitivity and specificity or the average of the fraction of correct labels that are predicted correctly (by the model) within each class. We used this metric because we observed that there was class imbalance within our datasets regardless of DILI type.MCC is particularly useful in datasets of different class distributions  because it considers all of the false and true positives and negatives. It is calculated from the confusion matrix of a model and its values range from +1 to −1, with +1 indicating a perfect classification, 0 indicating random classifications, and −1 indicating no relationships between the observed and predicted classes.In an attempt to improve the classification accuracy of our models, we used three ensemble voting approaches, namely soft voting, hard voting, and a weighted voting approach. These ensemble methods work best when there are varying algorithms of different strengths i.e. algorithms having varying underlying assumptions about the data, and when each one has reasonable predictive power . Using tHard voting, also known as majority voting, takes into account the predicted class labels of each classifier (or voter) . Voting The third approach to voting involves using a weight to skew predictions towards the most certain models. In our approach, we used the AUC of each classifier as a weighting parameter for the output probabilities. This was done to take into account that some classifiers might have better predictive power and should be given preference in determining the outcome of the voting. To weigh each probability, we multiplied the probabilities of each predicted class by the AUC and divided this by one subtracted from the weight, that is, the AUC of that model. Afterward, weighted probabilities were treated just as in soft voting: by taking the average of all resulting weighted probabilities belonging to each class. The class label with the higher average was taken as the predicted class for that observation. Therefore, the predicted class, The performance of FAERS data in predicting each of the DILI types can be seen in the bar plots in The top three models built using TOX21 data (using the AUC as the criterion) were evaluated on the test set . Using tSimilarly to how the FAERS data was handled, we selected the top three performing models built using MOLD2 data in each category (resampled or non-resampled) to predict the DILI class of the test data . Models Cellular RNA expression levels in the form of microarray data have been previously investigated for their ability to predict DILI with limited predictive power . In the The models built using the merged expression signatures with the highest AUC from the training data were evaluated on the test set. The training and test results are summarized in the bar plots in Since the top three individual models did not perform well on the test set , we askeCAMDA 2020 was a collaborative challenge to establish predictive models of DILI using gene expression data as well as a combination of data from clinically reported events, drug structure, and toxicology. In our study, we evaluated the predictability of these datasets on four DILI types, namely, DILI1 (severity score ≥6), DILI3 , DILI5 (negative control), and DILI6 (positive control). These datasets included gene expression/perturbation data on six cell lines , concentration-response or toxicology information, 2D molecular descriptors of the drug structure, and reported adverse events. To assess the predictive abilities of these datasets, we used various traditional machine learning algorithms. For non-gene expression datasets, we corrected the imbalance issue using well-known techniques like SMOTE, ROSE, and upsampling the minority class.While CAMDA previously approached predicting DILI, there have been significant improvements in the data provided and scopes of the challenge each year. In 2018 the challenge data only included microarray expression data from non-liver relevant cell lines on 276 compounds with a binary DILI classification. Published results from the 2018 challenge indicate limited success from both deep learning and soft voting approaches which achieved a maximum accuracy of 0.7 and MCC values < 0.2 . When thIn our study, we developed models with gene expression data using individual cell lines, as well as a merging of these datasets. Each cell line dataset did not include all the drugs thereby reducing the size of the training data and making it difficult to evaluate each of them on the test set. Therefore, we merged these datasets into one expression signature across cell types. Further, we selected the 100, 250, 500, and 1,000 most upregulated and downregulated genes as an arbitrarily signature cutoff of the most perturbed genes by drug treatment. However, our approach failed to capture predictive differences between the positive and negative classes in each DILI type. Although we achieved an accuracy of 0.67 for DILI1 (on the test set), a sensitivity of 0.38 showed that our models were not learning the positive classes well enough. Usually, this problem is due to not having sufficient training examples for a particular class. In contrast, we could obtain specificity as high as 0.95, showing that the model could learn the negative classes well since there were more DILI negative drugs in the training set. One challenge we had was that the training set was perhaps too small to be further split into a training and validation set. However, machine learning algorithms benefit most from having sufficient examples. For some datasets such as the gene expression datasets, we did not have access to information on all 617 drugs, which reduced the size of the training data. Besides, the training data were largely unbalanced . For insIn summary, our study suggests that currently available data, including mRNA quantification, molecular descriptors, clinically reported events, and toxicology profiles, may be inadequate to capture important information enough to separate DILI classes in real-world scenarios. Also, larger datasets may be needed to encourage the application of deep learning algorithms which typically do better with bigger data. We also suggest an additional focus or challenge to predict biomarkers specific for DILI using various–omics data, for instance, single-cell data and metabolomics signatures.