In this paper, we propose to inject latent entity type information into the entity embeddings by modeling the immediate context surrounding the mention. Specifically, we apply pre-trained BERT [1]} to represent the entity context and build a shared entity representation by aggregating all the entity contexts linking to the same entity via average pooling. Pre-trained BERT models naturally fit our purpose to represent the entity context surrounding the [MASK] token as it is trained with masked language model objective.
What's more, we integrate a BERT-based entity similarity feature into the local model of [2]} ganea2017deep to better capture entity type information. This can leverage both the pre-trained entity embeddings from BERT and the domain adaption capability of BERT via fine-tuning.
