We used the same input and output representations, i.e., the embedding and positional encoding, and the same loss objective, i.e., masked LM prediction and next sentence prediction, from the BERT paper <cite>(Devlin et al., 2018)</cite> .