Off-manifold samples can improve the uncertainty estimation in close OOD regions.
However, the generalization of adversarial samples relies on the diversity of the features of the training data.
[1]} report that models trained on CIFAR-10 can generate better adversarial examples for regularization than models trained on SVHN [2]}. Because CIFAR-10 contains more diverse features than SVHN, a dataset of only street numbers.
Our empirical observations find that off-manifold samples can help when combined with pre-trained transformers. However, it does not provide significant improvement in vanilla GRUs/ LSTMs.
This is consistent with the empirical study [3]} where pre-trained transformers outperform vanilla models in generalization towards OOD regions. The embeddings of pre-trained transformers contain rich features that benefit the generated adversarial examples.
Thus following [4]}, we evaluate off-manifold regularization on BERT [5]}.
<FIGURE>