The language model proposed by [1]} utilizes memorization of the training data to decrease the perplexity.
In this approach, the authors first compute representations of all sentence prefixes (keys) and store them together with the following word (values). They then use this for the next word prediction by softmaxing negative \(L^2\)  distancesThe negative \(L^2\)  distance is used instead of the common inner product as the similarity measure.
This was shown empirically by the authors to work better. of 1024 neighbours.
The representations (1024 dimensions) are the output of the last self-attention layer of the trained Transformer-based language model [2]}, [3]}.
The importance of the retrieved artefact in the output is determined by the manually set hyperparameter \(\lambda \in [0,1]\) , resulting in linear (convex) interpolation.
