A few recent methods show that it is possible to train exclusively with synthetic data and apply the trained network to the real world [1]}, [2]}. But for objects with mismatch between simulation and real world, these methods tend to fail, such as mismatch in texture materials or lighting. Meanwhile, some efforts are devoted to collecting large-scale real-world dataset for 6D pose estimation [3]}, [4]}. These methods use a moving camera around a static scene so that the pose labels for a video sequence can be generated with labeling one frame and tracking the camera ego-motion [3]} or labeling the 3D reconstructed scene [4]}. Although tracking camera motion simplifies the labeling process, these methods require human to arrange the scene, collect video sequences of objects, and initialize the systems, which is difficult to scale to a large number of scenes. In this work, we employ robots to automatically arrange scenes and generate accurate 6D pose annotations without pose initialization from human.
