Since the advent of the well-known BERT models [1]}, it has been applied successfully to and has achieved state-of-the-art performance on many NLP tasks.
The main challenges which the entity linking task has over other tasks e.g. sentence classification, named entity recognition, where BERT has been applied are: (1) a very large label space, i.e. every mention has many target entities and (2) the zero-shot nature of the entity linking task. Training label embeddings from a small labeled dataset could not generalize to cover unseen entities in test time. To tackle this problem, we introduce a novel method to build entity embeddings from BERT by modeling the immediate context of an entity.
