Model: 
Figure REF  (Left) shows the architecture of our model. We use a ResNet [1]} based architecture as our backbone model to process the 3D voxel input.
We use [2]} to adapt the ResNet model to operate on 3D input.
The output of the model is projected down to \(\mathbb {R}^{K=256}\)  using a linear projection.
We refer to the output of the linear projection as the relation embedding \(r(o_i, o_j) = f_\text{rel}(X)\) , where \(o_i\) , \(o_j\)  denotes the anchor and referrant object respectively and \(X\)  is the perceptual input with the object pair.
The projected relation embedding \(r(o_i, o_j)\)  is then concatenated with the action input \(a\) . We create \(a\)  by combing the action vector of size \(\mathbb {R}^3\)  with a one-hot label of size \(\mathbb {R}^2\) , this label is used to differentiate between fixed and adaptive perturbation actions.
This combined representation is then used to predict a set of outputs which correspond to the observed effects of this action.
<FIGURE>