Our proposed solution combines representation learning, planning, and online fine-tuning.
The key intuition is that, if we can learn a suitable representation of states and goals that generalizes effectively across environments, then we can plan subgoals in this representation space to solve long-horizon tasks, and also leverage it to help finetune the goal-conditioned policies on the new task online.
Core to this planning procedure is the use of affordance models [1]}, [2]}, which predict potentially reachable states that can serve as subgoal proposals for the planning process.
Good state representations are necessary for this process: (1) as inputs and outputs for the affordance model, which must generalize effectively across tasks and domains (since if the affordance model doesn't generalize, it can't provide guidance for policy fine-tuning); (2) as inputs into the policy, so as to facilitate rapid adaptation; (3) as measures of state proximity for use as reward functions for fine-tuning the policy, since well-shaped rewards are essential for rapid online training but notoriously difficult to obtain without manual engineering.
