World Value Functions: Knowledge Representation for Learning and Planning
We propose world value functions (WVFs), a type of goal-oriented general
value function that represents how to solve not just a given task, but any
other goal-reaching task in an agent's environment. This is achieved by
equipping an agent with an internal goal space defined as all the world states
where it experiences a terminal transition. The agent can then modify the
standard task rewards to define its own reward function, which provably drives
it to learn how to achieve all reachable internal goals, and the value of doing
so in the current task. We demonstrate two key benefits of WVFs in the context
of learning and planning. In particular, given a learned WVF, an agent can
compute the optimal policy in a new task by simply estimating the task's reward
function. Furthermore, we show that WVFs also implicitly encode the transition
dynamics of the environment, and so can be used to perform planning.
Experimental results show that WVFs can be learned faster than regular value
functions, while their ability to infer the environment's dynamics can be used
to integrate learning and planning methods to further improve sample
efficiency.