Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning
Recent progress in deep model-based reinforcement learning allows agents to
be significantly more sample efficient by constructing world models of
high-dimensional environments from visual observations, which enables agents to
learn complex behaviours in summarized lower-dimensional spaces. Reusing
knowledge from relevant previous tasks is another approach for achieving better
data-efficiency, which becomes especially more likely when information of
multiple previously learned tasks is accessible. We show that the simplified
representations of environments resulting from world models provide for
promising transfer learning opportunities, by introducing several methods that
facilitate world model agents to benefit from multi-source transfer learning.
Methods are proposed for autonomously extracting relevant knowledge from both
multi-task and multi-agent settings as multi-source origins, resulting in
substantial performance improvements compared to learning from scratch. We
introduce two additional novel techniques that enable and enhance the proposed
approaches respectively: fractional transfer learning and universal feature
spaces from a universal autoencoder. We demonstrate that our methods enable
transfer learning from different domains with different state, reward, and
action spaces by performing extensive and challenging multi-domain experiments
on Dreamer, the state-of-the-art world model based algorithm for visual
continuous control tasks.