Dropout: Dropout is a widely used Method to reduce overfitting Srivastava et al. (2014). We compared with Embedding Dropout which dropout each dimension of the token embedding. For fair comparison, we only add it to the input token.