We introduce an attention mechanism to improve feature extraction for deep active learning (AL) in the semi-supervised setting. The proposed attention mechanism is based on recent methods to visually explain predictions made by DNNs. We apply the proposed explanation-based attention to MNIST and SVHN classification. The conducted experiments show accuracy improvements for the original and class-imbalanced datasets with the same number of training examples and faster long-tail convergence compared to uncertainty-based methods. Deep active learning (AL) minimizes the number of expensive annotations needed to train DNNs by selecting a subset of relevant data points from a large unlabeled dataset BID7 . This subset is annotated and added to the training dataset in a single pool of data points or, more often, in an iterative fashion. The goal is to maximize prediction accuracy while minimizing the product of pool size × number of iterations. A proxy for this goal could be the task of matching feature distributions between the validation and the AL-selected training datasets.In density-based AL approaches, data selection is typically performed using a simple L 2 -distance metric BID10 . The image retrieval field BID17 has advanced much further in this area. For example, recent state-of-the-art image retrieval systems are based on DNNbased feature extraction BID0 with attention mechanisms BID8 . The latter estimates an attention mask to weight importance of the extracted features and it is trained along with the feature extraction.Inspired by this, we employ image retrieval techniques and propose a novel attention mechanism for deep AL. Unlike supervised self-attention in BID8 BID14 , our attention mechanism is not trained with the model. It relies on recent methods to generate visual explanations and to attribute feature importance values BID13 . We show the effectiveness of such explanation-based attention (EBA) mechanism for AL when combined with multi-scale feature extraction on a number of image classification datasets. We also conduct experiments for distorted class-imbalanced training data which is a more realistic assumption for unlabeled data. We applied recent image retrieval feature-extraction techniques to deep AL and introduced a novel EBA mechanism to improve feature-similarity matching. First feasibility experiments on MNIST and SVHN datasets showed advantages of EBA to improve density-based AL. Rather than performing AL for the well-picked training datasets, we also considered more realistic and challenging scenarios with class-imbalanced training collections where the proposed method emphasized the importance of additional feature supervision. In future research, EBA could be evaluated with other types of data distortions and biases: within-class bias, adversarial examples, etc. Furthermore, such applications as object detection and image segmentation may benefit more from EBA because multiscale attention can focus on spatially-important features.
We introduce an attention mechanism to improve feature extraction for deep active learning (AL) in the semi-supervised setting.
{"src": "scitldr-AIC_train", "paper_id": "SyxKiVmedV"}