As for tokens, the model takes an input sentence as a sequence of word tokens encoded through ComBERT, as introduced in [1]}. As shown in Figure REF , ComBERT is produced from BERT-based domain adaptive pre-training on a large collection of commodity news on the task of masked language modeling. It was trained on a collection of commodity news extracted from www.investing.com, the same source as the CrudeOilNews corpus. The idea is to adapt BERT, which was trained on English Wikipedia and Brown corpus [2]} to the finance and economics domains via Domain Adaptive Pre-training. In [1]}, it is shown that ComBERT produced better event extraction results than BERT and other generic language models. This is consistent with language models produced through domain adaptive pre-training on in-domain corpus such as SciBERT [4]} and BioBERT [5]}. All event extraction tasks described later use ComBERT embeddings and models are trained via fine-tuning from ComBERT.
