Contextualized word embeddings such as ELMo [1]}, BERT [2]}, and SpanBERT [3]} can be useful for relation extraction. These language models are trained on large corpora and can capture the contextual meaning of words in their vector representations. All neural models that are proposed for relation extraction use word representations such as Word2Vec [4]} or GloVe [5]} in their word embedding layer. Contextualized embeddings can be added in the embedding layer of the relation extraction models to improve their performance further. The SpanBERT model shows significant improvement in performance on the TACRED dataset. [3]} replaced the entity 1 token with its type and SUBJ such as PER-SUBJ and entity 2 token with its type and OBJ such as LOC-OBJ in the sentences to train the model. Finally, they used a linear classifier on top of the CLS token vector to find the relation.
