Sepsis is a leading cause of mortality in intensive care units (ICUs) and costs hospitals billions annually. Treating a septic patient is highly challenging, because individual patients respond very differently to medical interventions and there is no universally agreed-upon treatment for sepsis. Understanding more about a patient's physiological state at a given time could hold the key to effective treatment policies. In this work, we propose a new approach to deduce optimal treatment policies for septic patients by using continuous state-space models and deep reinforcement learning. Learning treatment policies over continuous spaces is important, because we retain more of the patient's physiological information. Our model is able to learn clinically interpretable treatment policies, similar in important aspects to the treatment policies of physicians. Evaluating our algorithm on past ICU patient data, we find that our model could reduce patient mortality in the hospital by up to 3.6 over observed clinical policies, from a baseline mortality of 13.7 . The learned treatment policies could be used to aid intensive care clinicians in medical decision making and improve the likelihood of patient survival.
Optimal sepsis treatment strategy was tackled most recently by @cite_3 , using a discretized state and action-space to deduce optimal treatment policies for septic patients. Their work applied on-policy SARSA learning to fit an action-value function to the physician policy and value-iteration techniques to find an optimal policy . The optimal policy was then evaluated by comparing the Q-values that would have been obtained following chosen actions to the Q-values obtained by the physicians. We reproduce a similar model as our baseline, using related data pre-processing and clustering techniques. We additionally build on this approach by extending the results to the continuous domain, where policies are learned directly from the physiological state data, without discretization. We also propose a novel evaluation metric, different from ones used in @cite_3 . We focus on in-hospital mortality instead of 90-day mortality (used in @cite_3 ) because of the other unobserved factors that could affect mortality in a 3-month timeframe.
@cite_3
2619989803 2768978543 2896893468 1999351703
Sepsis is a leading cause of mortality in intensive care units (ICUs) and costs hospitals billions annually. Treating a septic patient is highly challenging, because individual patients respond very differently to medical interventions and there is no universally agreed-upon treatment for sepsis. Understanding more about a patient's physiological state at a given time could hold the key to effective treatment policies. In this work, we propose a new approach to deduce optimal treatment policies for septic patients by using continuous state-space models and deep reinforcement learning. Learning treatment policies over continuous spaces is important, because we retain more of the patient's physiological information. Our model is able to learn clinically interpretable treatment policies, similar in important aspects to the treatment policies of physicians. Evaluating our algorithm on past ICU patient data, we find that our model could reduce patient mortality in the hospital by up to 3.6 over observed clinical policies, from a baseline mortality of 13.7 . The learned treatment policies could be used to aid intensive care clinicians in medical decision making and improve the likelihood of patient survival. Sepsis is a leading cause of mortality in intensive care units and costs hospitals billions annually. Treating a septic patient is highly challenging, because individual patients respond very differently to medical interventions and there is no universally agreed-upon treatment for sepsis. In this work, we propose an approach to deduce treatment policies for septic patients by using continuous state-space models and deep reinforcement learning. Our model learns clinically interpretable treatment policies, similar in important aspects to the treatment policies of physicians. The learned policies could be used to aid intensive care clinicians in medical decision making and improve the likelihood of patient survival. Sepsis is the third leading cause of death worldwide and the main cause of mortality in hospitals1–3, but the best treatment strategy remains uncertain. In particular, evidence suggests that current practices in the administration of intravenous fluids and vasopressors are suboptimal and likely induce harm in a proportion of patients1,4–6. To tackle this sequential decision-making problem, we developed a reinforcement learning agent, the Artificial Intelligence (AI) Clinician, which extracted implicit knowledge from an amount of patient data that exceeds by many-fold the life-time experience of human clinicians and learned optimal treatment by analyzing a myriad of (mostly suboptimal) treatment decisions. We demonstrate that the value of the AI Clinician’s selected treatment is on average reliably higher than human clinicians. In a large validation cohort independent of the training data, mortality was lowest in patients for whom clinicians’ actual doses matched the AI decisions. Our model provides individualized and clinically interpretable treatment decisions for sepsis that could improve patient outcomes. Objective. —To develop customized versions of the Simplified Acute Physiology Score II (SAPS II) and the 24-hour Mortality Probability Model II (MPM II24) to estimate the probability of mortality for intensive care unit patients with early severe sepsis. Design and Setting. —Logistic regression models developed for patients with severe sepsis in a database of adult medical and surgical intensive care units in 12 countries. Patients. —Of 11 458 patients in the intensive care unit for at least 24 hours, 1130 had severe sepsis based on criteria of the American College of Chest Physicians and the Society of Critical Care Medicine (systemic inflammatory response syndrome in response to infection, plus hypotension, hypoperfusion, or multiple organ dysfunction). Results. —In patients with severe sepsis, mortality was higher (48.0 vs 19.6 among other patients) and 28-day survival was lower. The customized SAPS II was well calibrated (P=.92 for the goodness-of-fit test) and discriminated well (area under the receiver operating characteristic [ROC] curve, 0.78). Performance in the validation sample was equally good (P=.85 for the goodness-of-fit test; area under the ROC curve, 0.79). The customized MPM II24was well calibrated (P=.92 for the goodness-of-fit test) and discriminated well (area under the ROC curve, 0.79). Performance in the validation sample was equally good (P=.52 for the goodness-of-fit test; area under the ROC curve, 0.75). The models are independent of each other; either can be used alone to estimate the probability of mortality of patients with severe sepsis. Conclusions. —Customization provides a simple technique to apply existing models to a subgroup of patients. Accurately assessing the probability of hospital mortality is a useful adjunct for clinical trials. (JAMA. 1995;273:644-650)