@inproceedings{
davydov2023retrieving,
title={Retrieving \$k\$-Nearest Memories with Modern Hopfield Networks},
author={Alexander Davydov and Sean Jaffe and Ambuj Singh and Francesco Bullo},
booktitle={Associative Memory {\&} Hopfield Networks in 2023},
year={2023},
url={https://openreview.net/forum?id=bNBMnQXRJU}
}
Modern continuous Hopfield networks (MCHNs) are a variant of Hopfield networks that have greater storage capacity and have been shown to have connections to the attention mechanism in transformers. In this paper, we propose a variant of MCHNs, which we call k-Hopfield layers, which is the first Hopfield-type network that retrieves the k-nearest memories to a given input. k-Hopfield layers are differentiable and may serve as (i) a soft approach to k-nearest neighbors, (ii) an augmented form of memory in deep learning architectures and (iii) an alternative to multihead attention in transformers. We empirically demonstrate that increasing k aids in correctly reconstructing a corrupted input. We show that using a k-Hopfield layer as a replacement to multihead attention demonstrates comparable performance in small vision transformers while requiring fewer parameters.
Retrieving k-Nearest Memories with Modern Hopfield Networks
Alexander Davydov Sean Jaffe Ambuj Singh Francesco Bullo