For multiple NLP tasks and primarily language modelling, the BERT [1]}, GPT-2 [2]}, GPT-3 [3]} and T5 [4]} models have seen great success.
Their performance is, however, limited on rare or unseen entities [5]}, [6]} and knowledge-intensive NLP tasks [7]}.
As a consequence, they perform poorly on the task of fact-aware language modelling, where the words to be predicted are named entities [5]}.
