We begin with the pre-trained BERT base and BERT large models, which respectively represent the normal and large model variants <cite>(Devlin et al., 2018)</cite> . To adapt BERT for document classifica- Figure 1 . During fine-tuning, we optimize the entire model end-to-end, with the additional softmax classifier parameters W ∈ IR K×H , where H is the dimension of the hidden state vectors and K is the number of classes.