While the default inductive behavior of CRF is to provide one sequence (the most probable one), we instead use an \(n\) -best sequence method which uses the Viterbi algorithm to return for each sentence the top \(n\)  sequences with their probabilities. We then query the annotator to annotate \(b\)  number of sentences (equal to a pre-selected batch size) with the highest entropies [1]} calculated using the following equation:
\(nSE(m, s) = - \sum _{\hat{y} \in \hat{Y}} p_m(\hat{y} | s; \theta ) \ log \ p_m(\hat{y}|s;\theta )\) 
