@inproceedings{
chaudhry2023long,
title={Long Sequence Hopfield Memory},
author={Hamza Tahir Chaudhry and Jacob A Zavatone-Veth and Dmitry Krotov and Cengiz Pehlevan},
booktitle={Associative Memory {\&} Hopfield Networks in 2023},
year={2023},
url={https://openreview.net/forum?id=sYAm62gWbo}
}
Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Computational models of sequence memory have been proposed where recurrent Hopfield-like neural networks are trained with temporally asymmetric Hebbian rules. However, these networks suffer from limited sequence capacity (maximal length of the stored sequence) due to interference between the memories. Inspired by recent work on Dense Associative Memories, we expand the sequence capacity of these models by introducing a nonlinear interaction term, enhancing separation between the patterns. We derive novel scaling laws for sequence capacity with respect to network size, significantly outperforming existing scaling laws for models based on traditional Hopfield networks, verify these theoretical results with numerical simulation, and demonstrate their usefulness in overlapping patterns. Finally, we describe a biologically-plausible implementation, with connections to motor neuroscience.
Long Sequence Hopfield Memory
Hamza Tahir Chaudhry Jacob A Zavatone-Veth Dmitry Krotov Cengiz Pehlevan