Renaissance Robot: Optimal Transport Policy Fusion for Learning Diverse Skills
Deep reinforcement learning (RL) is a promising approach to solving complex
robotics problems. However, the process of learning through trial-and-error
interactions is often highly time-consuming, despite recent advancements in RL
algorithms. Additionally, the success of RL is critically dependent on how well
the reward-shaping function suits the task, which is also time-consuming to
design. As agents trained on a variety of robotics problems continue to
proliferate, the ability to reuse their valuable learning for new domains
becomes increasingly significant. In this paper, we propose a post-hoc
technique for policy fusion using Optimal Transport theory as a robust means of
consolidating the knowledge of multiple agents that have been trained on
distinct scenarios. We further demonstrate that this provides an improved
weights initialisation of the neural network policy for learning new tasks,
requiring less time and computational resources than either retraining the
parent policies or training a new policy from scratch. Ultimately, our results
on diverse agents commonly used in deep RL show that specialised knowledge can
be unified into a "Renaissance agent", allowing for quicker learning of new
skills.