2 Related work 2.1 BERT Our work focuses on improving the transformer architecture (Vaswani et al., 2017) , which motivated the recent breakthrough in language representation, BERT <cite>(Devlin et al., 2018)</cite> .