To estimate robustness of our predictions at the inference stage, we use a confidence metric, which equals \(1 - uncertainty\) , as introduced in [1]}.
The key idea is that a classifier can only be confident on a text example if it predicts the correct labels for texts with similar embeddings. If the genre predicted by a classifier is not the same for many texts in the neighbourhood of an original text, we cannot consider the classifier confident. To simulate variation in embeddings, we apply dropout with probability 0.1 to all of the model layers, including the embedding layer and the final dense classification layer [2]}. A softmax classifier returns a probability distribution of the possible text labels. Normally, the label with the maximal probability is considered the answer of a classifier. Application of dropout to the classifier disturbs the probability distribution. We perform dropout \(n\)  times, and thereby we generate the corresponding probability distributions \(p_1, ..., p_n\) . Then we pool them into single distribution \(\widehat{p}\) . The maximum value of probability in \(\widehat{p}\)  is called the confidence of prediction. The intuition of this metric is that if a classifier is confident in the predicted text label, it is unlikely to distribute the likelihood on other class labels significantly often. In our study, we use \(n=10\)  - the same as used by the authors of [1]}, since this provides a good balance between assessing the confidence value and the speed of computation.
