Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approach
In meta reinforcement learning (meta RL), an agent learns from a set of
training tasks how to quickly solve a new task, drawn from the same task
distribution. The optimal meta RL policy, a.k.a. the Bayes-optimal behavior, is
well defined, and guarantees optimal reward in expectation, taken with respect
to the task distribution. The question we explore in this work is how many
training tasks are required to guarantee approximately optimal behavior with
high probability. Recent work provided the first such PAC analysis for a
model-free setting, where a history-dependent policy was learned from the
training tasks. In this work, we propose a different approach: directly learn
the task distribution, using density estimation techniques, and then train a
policy on the learned task distribution. We show that our approach leads to
bounds that depend on the dimension of the task distribution. In particular, in
settings where the task distribution lies in a low-dimensional manifold, we
extend our analysis to use dimensionality reduction techniques and account for
such structure, obtaining significantly better bounds than previous work, which
strictly depend on the number of states and actions. The key of our approach is
the regularization implied by the kernel density estimation method. We further
demonstrate that this regularization is useful in practice, when `plugged in'
the state-of-the-art VariBAD meta RL algorithm.