Each pair of channels for a state \(s\)  consists of a depth image \(\mathbf {d}_s \in \mathbb {R}^{H \times W}\)  and a mask image \(\mathbf {m}_s \in \mathbb {R}^{H \times W}\) , both in top down view (we use “top down depth image” and “heightmap” interchangeably). \(\mathbf {d}_s\)  may be obtained by capturing the end effector with one or more depth cameras, recovering a point cloud, and projecting it into a predefined plane corresponding to the workspace [1]}. Multiple views may be necessary if there is heavy visual occlusion of the end effector.
