Other baselines not included in our experiments include Deep Ensemble [1]},
which average the softmax outputs of five models with different initialization. A recent empirical study [2]} proves that Deep Ensemble performs better than Dropout and Temperature Scaling under dataset shift of NLP tasks using LSTM [3]}. However, fine-tuning multiple pre-trained transformer models is computationally expensive. Besides, the advantage of our considered baseline OE over this method has been reported in [4]}. Therefore we do not consider this method as a baseline in our paper.
Another line of work, Stochastic Variational Bayesian Inference [5]}, [6]}, [7]} can be applied to CNN models but hard to be applied in other architectures such as LSTMs [2]}. [9]}, [10]} also prove the advantage of ENNs over multiple Stochastic Variational Bayesian Inference methods.
