SemanticFusion [1]} is one of the early works that combined the semantic segmentation labels from deep ConvNet with the dense scene geometry from a SLAM system. It incrementally integrates per-frame semantic segmentation predictions into a dense 3D map by probabilistically associating the 2D frames with the 3D map. This combination not only generates a map with useful semantic information, but also shows the integration with a SLAM system helps to enhance the single frame segmentation. The two modules are loosely coupled in SemanticFusion.
[2]} proposed a self-supervised network that predicts consistent semantic labels for a map, by imposing constraints on the consistency of semantic predictions in multiple views.
DA-RNN [3]} introduces recurrent models into semantic segmentation framework to learn the temporal connections over multiple view frames, producing more accurate and consistent semantic labelling for volumetric maps from KinectFusion [4]}.
Yet these methods provide no information on object instances, which means that they are not able to distinguish among different ojects from the same category.
