Performative Reinforcement Learning
We introduce the framework of performative reinforcement learning where the
policy chosen by the learner affects the underlying reward and transition
dynamics of the environment. Following the recent literature on performative
prediction~\cite{Perdomo et. al., 2020}, we introduce the concept of
performatively stable policy. We then consider a regularized version of the
reinforcement learning problem and show that repeatedly optimizing this
objective converges to a performatively stable policy under reasonable
assumptions on the transition dynamics. Our proof utilizes the dual perspective
of the reinforcement learning problem and may be of independent interest in
analyzing the convergence of other algorithms with decision-dependent
environments.
  We then extend our results for the setting where the learner just performs
gradient ascent steps instead of fully optimizing the objective, and for the
setting where the learner has access to a finite number of trajectories from
the changed environment. For both the settings, we leverage the dual
formulation of performative reinforcement learning and establish convergence to
a stable solution. Finally, through extensive experiments on a grid-world
environment, we demonstrate the dependence of convergence on various parameters
e.g. regularization, smoothness, and the number of samples.