Studies on domain-specific language representation models empirically show that target task performance can be improved, when in-domain data is used for pre-training [1]}, [2]}, [3]}. These publicly available domain-specific models are valuable to the NLP community. However, the selection of in-domain data usually resorts to intuition, which varies across NLP practitioners (Section ). According to [4]}, the context specific usage of language is affected by three factors: field (the subject matter being discussed), tenor (the relationship between the participants in the discourse and their purpose) and mode (communication medium, such as `spoken' or `written'). Generally, the selection of pre-training data in existing domain-specific models is mainly based on the field rather than the tenor. For example, BioBERT [2]} and SciBERT [3]} are both pre-trained on scholar articles, but on different fields (biology and computer science).
