Offline pre-training and online fine-tuning.
Offline reinforcement learning pre-trains the policy and value function on a dataset \(\mathcal {D}_{\text{offline}}\) , consisting of previously collected experiences \((s_t^i, a_t^i, r_t^i, s_{t+1}^i)\) , where \(i\)  and \(t\)  are the index of the trajectory and time step.
In this work, we assume \(\mathcal {D}_{\text{offline}}\)  does not include data of the target task.
To solve the target task, the pre-trained policy is then fine-tuned by exploring the environment and collecting online data \(\mathcal {D}_{\text{online}}\) .
Following [1]}, we first pre-train both the policy \(\pi \)  and the affordance model \(m\)  on \(\mathcal {D}_{\text{offline}}\) , and then use the subgoals planned using the affordance model to guide the online exploration of \(\pi \) .
In the online phase, we freeze the weight of \(m\)  and fine-tune \(\pi \)  on a combination of \(\mathcal {D}_{\text{offline}}\)  and \(\mathcal {D}_{\text{online}}\) .
While our contribution is orthogonal to the specific choice of the offline RL algorithm, we use implicit Q-learning [2]} which is well-suited to online fine-tuning using an expectile loss to construct value estimates and advantage weighted regression to extract a policy.
