To estimate the 3D rotation \(\mathbf {R}\)  and 3D translation \(\mathbf {t}\)  of objects in the scene, we utilize the PoseRBPF framework [1]}. PoseRBPF is a Rao-Blackwellized particle filter combined with a learned auto-encoder network [2]} for 6D object pose estimation. In PoseRBPF, each particle consists of a 3D translation and a full distribution over \(\mathbf {SO}(3)\) . A 3D translation can be represented by \((u, v, z)\) , where \((u, v)\)  is the object center in the image and \(z\)  is the distance of object center from the camera in 3D.
