We jointly pre-train the representation with reinforcement learning on the offline data as shown in Fig. REF .
The original IQL objective optimizes \(\mathcal {L}_\text{RL} = \mathcal {L}_{\pi } + \mathcal {L}_{Q} + \mathcal {L}_{V}\)  as described in [1]}.
To facilitate generalization of the pre-trained policy and the affordance model,
we would like to abstract away redundant domain-specific information from the learned representation. For this purpose,
we add a variational information bottleneck (VIB) [2]} to the reinforcement learning objective, which constrains the mutual information \(I(s_t; z_t)\)  and \(I(g; z_g)\)  between the state space and the representation space by a constant \(C\) . The joint training of \(\phi \) , \(\pi \) , \(V\)  and \(Q\)  can be written as an optimization problem using \(\theta \)  to denote the model parameters:
\(\max _{\theta }\hspace{5.0pt}\mathcal {L}_\text{RL}\qquad \text{s.t.}\hspace{5.0pt}I(g; z_g) \le C ,\quad I(s_t; z_t) \le C \text{, for $t = 0, ..., T$}\) 
