Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learning
Many robotic tasks are composed of a lot of temporally correlated sub-tasks
in a highly complex environment. It is important to discover situational
intentions and proper actions by deliberating on temporal abstractions to solve
problems effectively. To understand the intention separated from changing task
dynamics, we extend an empowerment-based regularization technique to situations
with multiple tasks based on the framework of a generative adversarial network.
Under the multitask environments with unknown dynamics, we focus on learning a
reward and policy from the unlabeled expert examples. In this study, we define
situational empowerment as the maximum of mutual information representing how
an action conditioned on both a certain state and sub-task affects the future.
Our proposed method derives the variational lower bound of the situational
mutual information to optimize it. We simultaneously learn the transferable
multi-task reward function and policy by adding an induced term to the
objective function. By doing so, the multi-task reward function helps to learn
a robust policy for environmental change. We validate the advantages of our
approach on multi-task learning and multi-task transfer learning. We
demonstrate our proposed method has the robustness of both randomness and
changing task dynamics. Finally, we prove that our method has significantly
better performance and data efficiency than existing imitation learning methods
on various benchmarks.