Transformers from an Optimization Perspective
Deep learning models such as the Transformer are often constructed by
heuristics and experience. To provide a complementary foundation, in this work
we study the following problem: Is it possible to find an energy function
underlying the Transformer model, such that descent steps along this energy
correspond with the Transformer forward pass? By finding such a function, we
can reinterpret Transformers as the unfolding of an interpretable optimization
process across iterations. This unfolding perspective has been frequently
adopted in the past to elucidate more straightforward deep models such as MLPs
and CNNs; however, it has thus far remained elusive obtaining a similar
equivalence for more complex models with self-attention mechanisms like the
Transformer. To this end, we first outline several major obstacles before
providing companion techniques to at least partially address them,
demonstrating for the first time a close association between energy function
minimization and deep layers with self-attention. This interpretation
contributes to our intuition and understanding of Transformers, while
potentially laying the ground-work for new model designs.