Since the mask channel in the end effector representation encodes relevant information to avoid this (methodinputs), collecting more training data may reduce this failure mode. However, this failure mode also relates to one important ability that we have not addressed in this work: task-oriented grasping [1]}. For example, the grasp TAE is optimized to successfully grasp the presented object/tool but does not consider how to grasp tools to maximize success in successive steps. Inspired by relevant methods [1]}, [3]}, we may be able to reformulate the target value to address this (replace or multiply the success label from the current action with that of the last action in the episode), but we leave this to future work.
