Mining Pixels: Weakly Supervised Semantic Segmentation Using Image Labels
We consider the task of learning a classifier for semantic segmentation using weak supervision, in this case, image labels specifying the objects within the image. Our method uses deep convolutional neural networks (CNNs) and adopts an Expectation Maximization (EM) based approach maintaining the uncertainty on pixel labels. We focus on the following three crucial aspects of the EM based approach: (i) initialization; (ii) latent posterior estimation (E step) and (iii) the parameter update (M step) We show that saliency and attention maps provide good cues to learn an initialization model and allows us to skip the bad local maximum to which EM methods are otherwise traditionally prone. In order to update the parameters, we propose minimizing the combination of the standard softmax loss and the KL divergence between the true latent posterior and the likelihood given by the CNN. We argue that this combination is more robust to wrong predictions made by the expectation step of the EM method. We support this argument with empirical and visual results. We additionally incorporate an approximate intersection over union (IoU) term into the loss function for better parameter estimation. Extensive experiments and discussions show that: (i) our method is very simple and intuitive; (ii) requires only image level labels; and (iii) consistently outperforms other weakly/semi supervised state of the art methods with a very high margin on the PASCAL VOC 2012 dataset.
ArXiv