There is a growing body of work on learning policies from broad and diverse datasets in control and robotics using offline RL [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}.
Learned offline policies can also be fine-tuned with additional online interaction [11]}, [12]}, [13]}, [14]}, [15]}, [16]}.
While most of these methods focus on learning from offline data of the same task, this paper aims to utilize data of diverse environments and tasks to facilitate fine-tuning for novel tasks.
A wide range of methods have been proposed for multi-task learning [17]}, [18]} and meta learning [19]}, [20]}. In contrast to these methods, we do not assume the task labels or explicit reward functions are provided in the offline data, instead adopting a goal-conditioned approach that can utilize diverse datasets without any explicit task labels or rewards.
Goal-conditioned RL [21]}, [22]} aims to learn policies that can flexibly solve multiple tasks by taking different goals as inputs.
Through goal relabeling [23]}, a goal-conditioned policy can be trained with self-supervision in absence of meticulously designed reward functions [24]}, [25]}, [26]}, [27]}, [28]}, [29]}, [30]}, [31]}, [32]}, [33]}.
To guide the exploration of goal-conditioned policy, several methods use goals produced by a goal generator or affordance model that learns to capture the distribution of reachable goals [34]}, [35]}, [14]}, [37]}. Compared to this prior work, we show that we can utilize broad datasets by utilizing representation learning within goal-conditioned RL. Moreover, by utilizing planning, we can significantly extend the capabilities of goal-conditioned RL.
Recent works have also proposed to plan sequences of subgoals for learning long-horizon tasks [38]}, [39]}, [40]}, [41]}.
These methods successfully improve the performance of goal-conditioned RL, but have significant shortcomings in utilizing broad datasets to learn novel tasks.
[38]} plans in the latent space of a variational autoencoder [43]} for goal-conditioned RL in a single environment, but does not handle diverse prior data.
Nonparametric methods [39]} utilize graph-based planning, which makes them difficult to adapt for learning unseen tasks from data where states visited for the new task have not been traversed.
Instead of generating and planning subgoals in the original state space, our method learns a representation of goals to facilitate generalization across domains.
A number of methods have been proposed for learning latent representations of states for control and robotics [45]}, [46]}, [47]}, [48]}, [49]}, [50]}, [51]}, [52]}, [53]}.
While most focus on learning compact representations for learning the policy or the dynamics model, our method learns representations in an offline goal-conditioned reinforcement learning setting and uses the learned representations to facilitate subgoal planning across domains.
[54]}, [55]} use a mutual information objective learns representation of the transition for selecting the goals to guide exploration.
Instead of proposing subgoals from previously visited states or a prior distribution, our method learns a generative model of the latent goals to plan subgoals for target tasks do not exist in the offline data.
