Following the previous work <cite>(Devlin et al., 2018</cite>; Yang et al., 2019; Liu et al., 2019a; Lan et al., 2019) , we evaluate our models on the General Language Understanding Evaluation (GLUE) benchmark and the Stanford Question Answering Dataset (SQuAD 1.1) (Rajpurkar et al., 2016) .