Real-world Relation Extraction (RE) tasks are challenging to deal with, either due to limited training data or class imbalance issues. In this work, we present Data Augmented Relation Extraction (DARE), a simple method to augment training data by properly finetuning GPT2 to generate examples for specific relation types. The generated training data is then used in combination with the gold dataset to train a BERT-based RE classifier. In a series of experiments we show the advantages of our method, which leads in improvements of up to 11 F1 score points compared to a strong baseline. Also, DARE achieves new state-of-the-art in three widely used biomedical RE datasets surpassing the previous best results by 4.7 F1 points on average. Relation Extraction (RE) is the task of identifying semantic relations from text, for given entity mentions in it. This task, along with Named Entity Recognition, has become increasingly important recently due to the advent of knowledge graphs and their applications. In this work, we focus on supervised RE (Zeng et al., 2014; Lin et al., 2016; Wu et al., 2017; Verga et al., 2018) , where relation types come from a set of predefined categories, as opposed to Open Information Extraction approaches that represent relations among entities using their surface forms (Banko et al., 2007; Fader et al., 2011) . RE is inherently linked to Natural Language Understanding in the sense that a successful RE model should manage to capture adequately well language structure and meaning. So, almost inevitably, the latest advances in language modelling with Transformer-based architectures (Radford et al., 2018a; Devlin et al., 2018; Radford et al., 2018b) have been quickly employed to also deal with RE tasks (Soares et al., 2019; Lin et al., 2019; Shi and Lin, 2019; Papanikolaou et al., 2019) . These recent works have mainly leveraged the discriminative power of BERT-based models to improve upon the state-of-the-art. In this work we take a step further and try to assess whether the text generating capabilities of another language model, GPT-2 (Radford et al., 2018b) , can be applied to augment training data and deal with class imbalance and small-sized training sets successfully. Specifically, given a RE task we finetune a pretrained GPT-2 model per each relation type and then use the resulting finetuned models to generate new training samples. We then combine the generated data with the gold dataset and finetune a pretrained BERT model (Devlin et al., 2018) on the resulting dataset to perform RE. We conduct extensive experiments, studying different configurations for our approach and compare DARE against two strong baselines and the stateof-the-art on three well established biomedical RE benchmark datasets. The results show that our approach yields significant improvements against the rest of the approaches. To the best of our knowledge, this is the first attempt to augment training data with GPT-2 for RE. In Table 1 we show some generated examples with GPT-2 models finetuned on the datasets that are used in the experiments (refer to Section 4). In the following, we provide a brief overview of related works in Section 2, we then describe our approach in Section 3, followed by our experimental results (Section 4) and the conclusions (Section 5). We have presented DARE, a novel method to augment training data in Relation Extraction. Given a gold RE dataset, our approach proceeds by finetuning a pre-trained GPT-2 model per relation type and then uses the finetuned models to generate new training data. We sample subsets of the synthetic data with the gold dataset to finetune an ensemble of RE classifiers that are based on BERT. On a series of experiments we show empirically that our method is particularly suited to deal with class imbalance or limited data settings, recording improvements up to 11 F1 score points over two strong baselines. We also report new state-of-the-art performance on three biomedical RE benchmarks. Our work can be extended with minor improvements on other Natural Language Understanding tasks, a direction that we would like to address in future work.
Data Augmented Relation Extraction with GPT-2
{"src": "scitldr-AIC_train", "paper_id": "rJedNwij2r"}