We explore active learning (AL) for improving the accuracy of new domains in a natural language understanding (NLU) system. We propose an algorithm called Majority-CRF that uses an ensemble of classification models to guide the selection of relevant utterances, as well as a sequence labeling model to help prioritize informative examples. Experiments with three domains show that Majority-CRF achieves 6.6 -9 relative error rate reduction compared to random sampling with the same annotation budget, and statistically significant improvements compared to other AL approaches. Additionally, case studies with human-in-the-loop AL on six new domains show 4.6 -9 improvement on an existing NLU system.
@cite_18 showed that AL is susceptible to the missed cluster effect when selection focuses only on examples around the existing decision boundary and misses important clusters of data that receive high confidence. They conclude that AL may produce a sub-optimal classifier compared to random sampling with large budget. To solve this problem @cite_17 proposed an AL algorithm that can balance exploitation (sampling around the decision boundary) and exploration (random sampling) by reallocating the annotation budget between the two. In our case, we iteratively select and annotate small batches of data that are used as feedback in subsequent selections, such that extensive exploration is not required. Moreover, we always dedicate part of the annotation budget to random sampling to avoid missed clusters, but in this paper we do not discuss the effects of the additional randomly sampled data.
@cite_18 @cite_17
2137061486 2123108597
In practical classification, there is often a mix of learnable and unlearnable classes and only a classifier above a minimum performance threshold can be deployed. This problem is exacerbated if the training set is created by active learning. The bias of actively learned training sets makes it hard to determine whether a class has been learned. We give evidence that there is no general and efficient method for reducing the bias and correctly identifying classes that have been learned. However, we characterize a number of scenarios where active learning can succeed despite these difficulties. Active machine learning algorithms are used when large numbers of unlabeled examples are available and getting labels for them is costly (e.g. requiring consulting a human expert). Many conventional active learning algorithms focus on refining the decision boundary, at the expense of exploring new regions that the current hypothesis misclassifies. We propose a new active learning algorithm that balances such exploration with refining of the decision boundary by dynamically adjusting the probability to explore at each step. Our experimental results demonstrate improved performance on data sets that require extensive exploration while remaining competitive on data sets that do not. Our algorithm also shows significant tolerance of noise.