RL algorithms are increasingly used in mobile health studies, for example, in studies to increase users' physical activity [1]}, [2]}, [3]}, [4]}, [5]}, to promote users' weight loss [6]}, and to help users' manage mental illness [7]}. Many of these works use contextual bandit algorithms, a popular choice for optimizing intervention delivery under real-world constraints discussed in [8]} and [2]}. Alternative algorithms include [3]} which uses a modified version of a posterior sampling contextual bandit algorithm, [5]} which uses an MDP framework and learns an action selection strategy offline, and [4]} which uses inverse RL to estimate a model of the reward and mixed-integer linear programming to select a 7-day schedule of personal step count goals for each user. Similar to this work, [3]} incorporated a term to approximate the delayed effects of the current action on future rewards. However, their approach did not involve explicitly developing a surrogate reward and they additionally did not allow the term modeling the delayed effects of actions to depend on previous user outcomes (only on previous actions). Finally, they also did not develop simulation environments with delayed effects to evaluate the design of their delayed effects of actions and do not provide explicit advice on how to choose the hyperparameters in their delayed effects term.
