In our system, a RGB-D camera is mounted to the hand of a robot manipulator, where images from different views can be collected by moving the robot arm (Fig. REF ). The system consists of a pose initialization module, a pose tracking module, a robot interaction module and a self-supervised training module (Fig. REF ). 1) Given a set of objects in the robot's workspace, the pose initialization module estimates the 6D poses of all objects from a single RGB-D image. To ensure accurate pose initialization, we evaluate the estimated poses and re-initialize the unsatisfied objects until all the poses are accurate. 2) Then the pose tracking module tracks the 6D poses of these objects when the robot arm moves to different predefined way points. This step also collects images with 6D object poses from different views, and saves the data in an online fashion. Both the pose initialization and pose tracking are based on the PoseRBPF framework using particle filtering [1]}. 3) After the data is saved, the robot interaction module controls the manipulator to either grasp or push one object in the scene, thereby generating a new scene. In this way, we can continuously collect data with different arrangements of objects. 4) The self-supervised training module fine-tunes the deep neural networks in the system with the saved data to boost its performance.
