In this paper, we propose a novel deep learning architecture for joint classification and localization of abnormalities in mammograms. We first assume a weakly supervised setting and present a new approach with data driven decisions. This novel network combines two learning branches with region-level classification and region ranking. The network provides a global classification of the image into multiple classes, such as malignant, benign or normal. Our method further enables the localization of abnormalities as global class discriminative regions in full mammogram resolution. Next, we extend this method to a semi-supervised setting that engages a small set of local annotations, using a novel architecture, and a multi-task objective function. We present the impact of the local annotations on several performance measures, including localization, to evaluate the cost effectiveness of lesion annotation effort. Our evaluation is made over a large multi-center mammography dataset of @math 3,000 mammograms with various findings. Experimental results demonstrate the capabilities and advantages of the proposed method over previous weakly-supervised strategies, and the impact of semi-supervised learning. We show that targeting the annotation of only 5 of the images can significantly boost performance.
Recent advances of deep learning have achieved remarkable performances in various computer vision tasks including weakly supervised object localization. Weakly supervised object localization is practically useful since it does not require fine-grained annotations. Current approaches overcome the difficulties of weak supervision via transfer learning from pre-trained models on large-scale general images such as ImageNet. However, they cannot be utilized for medical image domain in which do not exist such priors. In this work, we present a novel weakly supervised learning framework for lesion localization named as self-transfer learning (STL). STL jointly optimizes both classification and localization networks to help the localization network focus on correct lesions without any types of priors. We evaluate STL framework over chest X-rays and mammograms, and achieve significantly better localization performance compared to previous weakly supervised localization approaches.