The environment and tasks we introduce present challenging problems but also make some simplifications which should be addressed. For instance, target objects are spawned into the workspace out of thin air, one at a time before each action, which is only possible in simulation. In a more realistic setting, all objects of concern would be present from the beginning. For example, in RGB-Stacking [1]}, each episode begins with 3 color-coded objects in which one is only present as a distractor. For tool use, an interesting problem would be to present multiple potential tools and make the robot select the most appropriate one given the task.
