The most common approach of training domain-specific models is continue pre-training, which is used by BioBERT [1]}, ClinicalBERT [2]}, and BERTweet [3]}. Continue pre-training approach starts from an existing pre-trained model—usually pre-trained on large size generic data set—and continues training on a domain-specific corpus. The main advantage of continue pre-training is that they can inherit knowledge from language representation models pre-trained on generic data, and thus be considered as capturing both generic domain and domain-specific knowledge. However, we aim to investigate the impact of pre-training data, therefore, we use the learning from scratch approach, eschewing the potential impact of generic data. We follow the hyper-parameter setting in [4]}, shown in Table REF  to train the domain-specific models. Training of each model took four days using 1 Nvidia Tesla v100 GPU.
<TABLE>