Pursuing a full MDP-based RL algorithm involves modeling both state transition probabilities and the expected reward for each state and action to form a policy. As discussed in the introduction, due to the constrained setting of Oralytics, we do not expect to have enough data to effectively learn parameters for a full MDP-based algorithm. Thus, we modify a contextual bandit algorithm—specifically, we use a variant of linear posterior sampling [1]}. Posterior sampling involves using a Bayesian model for the mean reward and selection actions according to the posterior probability that each action is optimal. The Bayesian framework allows us to incorporate data from previous studies and domain knowledge into the prior distribution. Further, the actions are selected probabilistically, which facilitates after-study inference [2]}, [3]}.
