
where \(o\)  is the current observation, \(\psi \)  is the Spatial Interaction Encoder Network from Section , \(g\)  is the representative goal frame, \(m_i\)  is the length of sequence \(V^{\prime i}\)  and \(c\)  is a scaling factor. The scaling factor c is computed as the average distance between the first and final observation of all the training videos in the learned embedding space. Note, that the range of the learned reward is \((-\infty , 0]\) . Defining the reward function in this way gives us a dense reward because as the observed state gets closer and closer to the goal, the reward starts going down and approaches zero when the goal and current observation are close in embedding space. After constructing the learned reward, we can use it to train any standard RL algorithm. We note that, unlike previous approaches [1]}, [2]}, our method does not use any environment reward to improve performance, and instead relies solely on the learned reward, which our experiments demonstrate is sufficient for solving diverse robotic manipulation tasks.
