However, classical bandit algorithms, designed to optimize the immediate reward, are not equipped to account for the delayed effects of actions (i.e., the impact of the current intervention on a user's responsivity to future interventions). Inappropriate (e.g., untimely or too many) notification interruptions may cause user burden, which can annoy users and cause them to be unresponsive. Preventing user burden is crucial, especially in mobile health settings, which typically have high user dropout rates [1]}, [2]}. One option is to use RL algorithms that model a full Markov decision process (MDP), specifically by modeling how the action impacts the next state and the future rewards achievable when starting in that next state. However, because of the constrained setting of Oralytics, specifically, the highly-noisy outcomes and limited data per user, we do not expect to have enough data to
effectively learn the large number of parameters needed in a MDP-based algorithm. Moreover, MDP-based algorithms may not be as stable to run and update online as bandit algorithms. Bandit algorithms can also be interpreted as a form of discount regularization for a full MDP-based RL algorithm [3]}.
