Language model pre-training has attracted wide attention and fine-tuning on pre-trained language model has shown to be effective for improving many downstream natural language processing tasks. BERT [1]} obtained new state-of-the-art results on a broad spectrum of diverse tasks, offering pre-trained deep bidirectional representations which are conditioned on both left and right context in all layers, which is often followed by discriminative fine-tuning on each specific task, including passage re-ranking for open domain QA.
