With the collected images and estimated object poses, we can fine-tune the neural networks in our system to improve the pose estimation performance. Initially, both the network for segmentation and detection in PoseCNN and the auto-encoders used in PoseRBPF (see Fig. REF ) are trained only with synthetic data. As a result, the system cannot even segment some textureless objects in our experiments such as a red foam brick. Therefore, we employ the curriculum learning idea [1]} to bootstrap the system sequentially. We start with simple scenes that contain only one object. If the network cannot segment the object, we perform a global initialization in PoseRBPF, where particles are uniformly sampled in the image. This process collects training data with single objects in images. We can then fine-tune the segmentation network to make it segment all objects. Fine-tuning with single objects also enables the network to segment objects in cluttered scenes. In the next stage, we move to cluttered scenes with multiple objects, and collect training data with different object poses and occlusions to fine-tune the segmentation network and the auto-encoders. During fine-tuning, we mix synthetic and real data to prevent the networks from overfitting. We can iterate between data collection and fine-tuning. Because when the networks become better, the system is more efficient in collecting new data.
