The KNN-LM approach involves linearly interpolating the extended neural language model with the K-Nearest Neighbours in the pre-trained LM embedding space. Zhong et al. (2022) proposed different processing for three types of memories (local, long-term and external) and added training for in-batch tokens to KNN-LM. The proposed changes aim to improve the performance of the model. Unlike KNN-LM, which only uses memory units during training, TRIME Zhong et al. (2022) uses memory units during both testing and training. He et al. (2021) suggested that not all generated tokens need to be retrieved. Instead, a lightweight neural network can be trained to aid the KNN-LM in adaptive retrieval. Additionally, efficiency can be improved through database streamlining and dimension reduction. Alon et al. (2022) proposed RETOMATON, an unsupervised, weighted finite automaton built on top of the data store. RETOMATON is based on saving pointers between successive data store entries and clustering techniques. RETOMATON is more effective than ADAPTRETHe et al. (2021) in improving accuracy by utilizing remaining pointers during KNN retrieval. Even without KNN retrieval, interpolation operations can still be performed using the stored previous information in the pointers, unlike ADAPTRET which solely relies on the language model. Furthermore, RETOMATON is unsupervised, requiring no additional data for training, making it more data-efficient. Grave et al. (2016) proposed using continuous cache to improve the performance of KNN-LM. This involves storing past hidden activations and accessing them at the appropriate time by dot product with present hidden activation. Yogatama et al. (2021) utilise an extended short-term context by caching local hidden states and global long-term memory by retrieving a set of nearest-neighbour tokens at each time step. They also design a gating function to adaptively combine multiple sources of information for prediction. Compared to KNN-LM, this method uses dynamic weights and can handle cases where interpolation is not feasible, such as when the memory output is an image, video, or sound. Drozdov et al. (2022) proposed a method for adjusting the interpolation weights. The weights are dynamically adjusted based on the size of the region of overlap between the retrieved stored data and the assessment set, which reflects the quality of the retrieval.