During active learning, an effective stopping method allows users to limit the number of annotations, which is cost effective. In this paper, a new stopping method called Predicted Change of F Measure will be introduced that attempts to provide the users an estimate of how much performance of the model is changing at each iteration. This stopping method can be applied with any base learner. This method is useful for reducing the data annotation bottleneck encountered when building text classification systems.
In 2008, Laws and Schutze proposed two stopping methods called performance convergence and uncertainty convergence @cite_37 . Performance convergence stops when the estimation of accuracy of the unlabeled pool converges so that the gradient of performance estimates is below a certain threshold. Uncertainty convergence stops when the uncertainty of the last selected instance of the batch has an uncertainty measure below a minimum threshold. For support vector machines and other learners that do not provide probabilities of classification, only uncertainty convergence can be used.
@cite_37
1971731086
Active learning is a proven method for reducing the cost of creating the training sets that are necessary for statistical NLP. However, there has been little work on stopping criteria for active learning. An operational stopping criterion is necessary to be able to use active learning in NLP applications. We investigate three different stopping criteria for active learning of named entity recognition (NER) and show that one of them, gradient-based stopping, (i) reliably stops active learning, (ii) achieves nearoptimal NER performance, (iii) and needs only about 20 as much training data as exhaustive labeling.