ST-CoNAL: Consistency-Based Acquisition Criterion Using Temporal Self-Ensemble for Active Learning
Modern deep learning has achieved great success in various fields. However,
it requires the labeling of huge amounts of data, which is expensive and
labor-intensive. Active learning (AL), which identifies the most informative
samples to be labeled, is becoming increasingly important to maximize the
efficiency of the training process. The existing AL methods mostly use only a
single final fixed model for acquiring the samples to be labeled. This strategy
may not be good enough in that the structural uncertainty of a model for given
training data is not considered to acquire the samples. In this study, we
propose a novel acquisition criterion based on temporal self-ensemble generated
by conventional stochastic gradient descent (SGD) optimization. These
self-ensemble models are obtained by capturing the intermediate network weights
obtained through SGD iterations. Our acquisition function relies on a
consistency measure between the student and teacher models. The student models
are given a fixed number of temporal self-ensemble models, and the teacher
model is constructed by averaging the weights of the student models. Using the
proposed acquisition criterion, we present an AL algorithm, namely
student-teacher consistency-based AL (ST-CoNAL). Experiments conducted for
image classification tasks on CIFAR-10, CIFAR-100, Caltech-256, and Tiny
ImageNet datasets demonstrate that the proposed ST-CoNAL achieves significantly
better performance than the existing acquisition methods. Furthermore,
extensive experiments show the robustness and effectiveness of our methods.