Learning.
We use Implicit Q-Learning(IQL) [1]} as the underlying RL algorithm.
To avoid evaluating actions outside of the offline dataset, IQL performs a modified TD update by approximating an upper expectile of the distribution over values.
We use Adam optimizer with a learning rate of \(3 \times 10^{-4}\)  and a batch size of 128. The Lagrange multipliers in Eqn. REF  and Eqn. REF  are chosen through grid search and are set to be \(\alpha = 0.01\)  and \(\beta = 0.1\) .
During offline pre-training (as in Fig. REF ), we train the model on the previously collected datasets with only relabeled goals [2]}.
While during online fine-tuning, we use \(30\%\)  ground truth goals and \(70\%\)  relabeled goals.
In each batch, we sample 60% of the samples from the offline dataset and the rest 40% from the online replay buffer following the practice of [3]}.
We use 1 TITAN RTX GPU with 24GB of memory and 1 Intel Xeon Skylake 6130 CPUS with 48GB of memory during training.
The pre-training takes around 10 hours in total and the fine-tuning takes around 2 hours for each target task.
