Environments and tasks.
Our experiments are conducted in table-top manipulation environments using low-level visuomotor control, with observations and goals corresponding to RGB images captured by a third-person camera.
We use a real-world WidowX robot operating in kitchen-themed environments at a frequency of 5 Hz through 7-DoF control.
The action space is defined as the translation, rotation, and finger position of the gripper, without using any hard-coded motion primitive.
In each target task, the robot is asked to reach the goal in 200 steps by interacting with multiple objects unseen in the environment.
The target tasks include moving a metal pot from one stove to another (Task A), moving a colander onto the stove and picking and placing a stuffed toy into the pot (Task B), and placing a sushi on a plate and then dropping a knife into a pan in the sink (Task C).
We also use a simulated environment for additional ablations and experiments based on prior work [1]}.
The simulated target tasks require sequential interactions with the environment such as opening/closing the drawer, sliding the cylinder, and picking and placing the block.
To complete these tasks, the robot needs to reason about the objects' affordances and spatial relations from the visual inputs and strategically interact with the environment in a feasible sequential order.
