Texts from biomedical publications and electronic medical record have been used to pre-train BERT models for NLP task in this domain and showed considerable improvement in many downstream tasks (Lee et al., 2019;<cite> Alsentzer et al., 2019</cite>; Si et al., 2019) .