Recent studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when the BERT models are further pre-trained on in-domain data [1]}. We thus choose SciBERT [2]}, which is pre-trained on full text of scholarly articles about biology and computer science, and fine-tune it on the target NER task. In our preliminary experiments, we observe that SciBERT achieves significant better results than vanilla BERT [3]} and slightly better results than BioBERT [4]}.
