Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines
Online reinforcement learning (RL) algorithms are increasingly used to
personalize digital interventions in the fields of mobile health and online
education. Common challenges in designing and testing an RL algorithm in these
settings include ensuring the RL algorithm can learn and run stably under
real-time constraints, and accounting for the complexity of the environment,
e.g., a lack of accurate mechanistic models for the user dynamics. To guide how
one can tackle these challenges, we extend the PCS (Predictability,
Computability, Stability) framework, a data science framework that incorporates
best practices from machine learning and statistics in supervised learning (Yu
and Kumbier, 2020), to the design of RL algorithms for the digital
interventions setting. Further, we provide guidelines on how to design
simulation environments, a crucial tool for evaluating RL candidate algorithms
using the PCS framework. We illustrate the use of the PCS framework for
designing an RL algorithm for Oralytics, a mobile health study aiming to
improve users' tooth-brushing behaviors through the personalized delivery of
intervention messages. Oralytics will go into the field in late 2022.