After we quantify the similarity between source and target datasets, the next step is to investigate the impact of source data on pre-trained language representation models. We pre-train ELECTRA [1]}—a sample-efficient variant of BERT—on different sources separately, then observe how the effectiveness of these pre-trained models varies in different downstream NER datasets.
