The resources (word and entity embeddings) used to train the local context model of [1]} ganea2017deep are obtained from DeepEDhttps://github.com/dalab/deep-ed/. For each entity, we randomly sample at most 100 anchor contexts from WikipediaUsing the same Wikipedia dump (Feb 2014) as the one which [1]} ganea2017deep used to train their entity embeddings. to build the entity representation from BERT. We discard any articles appearing in WIKI dataset when building the entity representation from BERT.
We take the anchor context as the surrounding sentence where the mention lies and replace the mention with a single [MASK] token.
Each context is truncated to 128 tokens after WordPiece tokenization.
We use the PyTorch implementation of pre-trained BERT modelshttps://github.com/huggingface/pytorch-pretrained-BERT and choose the BERT-base-cased version. We adopt the Adam [3]} implemented by BERT with \(\beta _1=0.9\) , \(\beta _2=0.999\) . Empirically, we found that it is helpful to set parameters in BERT a small initial learning rate and not BERT related parameters a larger initial learning rate to avoid the whole model biasing toward the BERT feature and disregarding other model components.
In our experiments, pre-trained BERT model is fine-tuned with initial learning rate \(10^{-5}\)  whereas not BERT related parameters are trained with \(10^{-3}\) .
Similar learning rate usage can be found in the recent work by [4]}.
Similar to [1]} ganea2017deep, all the entity embeddings are fixed during fine-tuning. We randomly initialize the not BERT related parameters using Gaussian distribution \(\mathcal {N}(0.0, 0.02)\)  and the bias term is zeroed.
