Deconfounding Actor-Critic Network with Policy Adaptation for Dynamic Treatment Regimes
Despite intense efforts in basic and clinical research, an individualized
ventilation strategy for critically ill patients remains a major challenge.
Recently, dynamic treatment regime (DTR) with reinforcement learning (RL) on
electronic health records (EHR) has attracted interest from both the healthcare
industry and machine learning research community. However, most learned DTR
policies might be biased due to the existence of confounders. Although some
treatment actions non-survivors received may be helpful, if confounders cause
the mortality, the training of RL models guided by long-term outcomes (e.g.,
90-day mortality) would punish those treatment actions causing the learned DTR
policies to be suboptimal. In this study, we develop a new deconfounding
actor-critic network (DAC) to learn optimal DTR policies for patients. To
alleviate confounding issues, we incorporate a patient resampling module and a
confounding balance module into our actor-critic framework. To avoid punishing
the effective treatment actions non-survivors received, we design a short-term
reward to capture patients' immediate health state changes. Combining
short-term with long-term rewards could further improve the model performance.
Moreover, we introduce a policy adaptation method to successfully transfer the
learned model to new-source small-scale datasets. The experimental results on
one semi-synthetic and two different real-world datasets show the proposed
model outperforms the state-of-the-art models. The proposed model provides
individualized treatment decisions for mechanical ventilation that could
improve patient outcomes.