First, the interpretations of states (e.g., the sets of categories describing the person and the notification) in the agent's state space are pre-defined by the system designers. Second, the interpretations of actions (e.g., sending and dropping notifications) in the action space are defined by system designers. Third, rewards are chosen by system designers and typically try to maximize things like click through rate, dwell time, or business revenue, which may have little to no value from the data subject's perspective. Fourth, any constraints on the set of allowable policies from within the policy space are set by system designers. Taken together, these problems cast doubt on aspirations of personalization by RL systems, if we are to take the person and his epistemic capacities seriously. Few if any of the design parameters capture what the user would reflectively endorse as the interpretation of an action or state, or the selection of the reward to be maximized. These tensions suggest an area ripe for research on how a person's moral values can influence choices such as the reward and criteria for allowable policies. We also see promise in connecting “teacher advice” for safe exploration [1]} by RL with persons' moral values and personal identities.
