@inproceedings{
hoover2023energy,
title={Energy Transformer},
author={Benjamin Hoover and Yuchen Liang and Bao Pham and Rameswar Panda and Hendrik Strobelt and Duen Horng Chau and Mohammed J Zaki and Dmitry Krotov},
booktitle={Associative Memory {\&} Hopfield Networks in 2023},
year={2023},
url={https://openreview.net/forum?id=TNw5KrKppB}
}
Our work combines aspects of three promising paradigms in
    machine learning, namely, attention mechanism, energy-based models, and
    associative memory. Attention is the power-house driving modern deep
    learning successes, but it lacks clear theoretical foundations. Energy-based
    models allow a principled approach to discriminative and generative
    tasks, but the design of the energy functional is not straightforward. At
    the same time, Dense Associative Memory models or Modern Hopfield Networks
    have a well-established theoretical foundation, and allow an intuitive
    design of the energy function. We propose a novel architecture, called the
    Energy Transformer (or ET for short), that uses a sequence of attention
    layers that are purposely designed to minimize a specifically engineered
    energy function, which is responsible for representing the relationships
    between the tokens. In this work, we introduce the theoretical foundations
    of ET, explore its empirical capabilities using the image completion task,
    and obtain strong quantitative results on the graph anomaly detection and
    graph classification tasks.
Energy Transformer
Benjamin Hoover Yuchen Liang Bao Pham Rameswar Panda Hendrik Strobelt Duen Horng Chau Mohammed J Zaki Dmitry Krotov