@inproceedings{
cabannes2023associative,
title={Associative Memories with Heavy-Tailed Data},
author={Vivien Cabannes and Elvis Dohmatob and Alberto Bietti},
booktitle={Associative Memory {\&} Hopfield Networks in 2023},
year={2023},
url={https://openreview.net/forum?id=guPW3ACk2L}
}
Learning arguably involves the discovery and memorization of abstract rules.
But how associative memories appear in transformer architectures optimized with gradient descent algorithms?
We derive precise scaling laws for a simple input-output associative memory model with respect to parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms.
We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations.
Associative Memories with Heavy-Tailed Data
Vivien Cabannes Elvis Dohmatob Alberto Bietti