Provably Sample-Efficient RL with Side Information about Latent Dynamics
We study reinforcement learning (RL) in settings where observations are
high-dimensional, but where an RL agent has access to abstract knowledge about
the structure of the state space, as is the case, for example, when a robot is
tasked to go to a specific room in a building using observations from its own
camera, while having access to the floor plan. We formalize this setting as
transfer reinforcement learning from an abstract simulator, which we assume is
deterministic (such as a simple model of moving around the floor plan), but
which is only required to capture the target domain's latent-state dynamics
approximately up to unknown (bounded) perturbations (to account for environment
stochasticity). Crucially, we assume no prior knowledge about the structure of
observations in the target domain except that they can be used to identify the
latent states (but the decoding map is unknown). Under these assumptions, we
present an algorithm, called TASID, that learns a robust policy in the target
domain, with sample complexity that is polynomial in the horizon, and
independent of the number of states, which is not possible without access to
some prior knowledge. In synthetic experiments, we verify various properties of
our algorithm and show that it empirically outperforms transfer RL algorithms
that require access to "full simulators" (i.e., those that also simulate
observations).