Our method unifies gripper and tool use by representing them in the same input space (methodinputs). Inspired by prior work [1]}, [2]}, we take advantage of fully convolutional architectures and cross-convolutions which provide an inductive bias that leads to more efficient learning (methodarch). The model takes as input an image representation of the end effector (“end effector” can refer to the gripper and/or the grasped tool). and the scene (the workspace) and outputs a dense action-value map in which each pixel value represents the likelihood of task success with a corresponding action. An overview of the model is visualized in fig:arch. We also describe how we collected the data (methoddata) and trained the model (methodtrain) in an offline fashion, enabling easier benchmarking for future work.
<FIGURE>