Transformer with Fourier Integral Attentions
Multi-head attention empowers the recent success of transformers, the
state-of-the-art models that have achieved remarkable success in sequence
modeling and beyond. These attention mechanisms compute the pairwise dot
products between the queries and keys, which results from the use of
unnormalized Gaussian kernels with the assumption that the queries follow a
mixture of Gaussian distribution. There is no guarantee that this assumption is
valid in practice. In response, we first interpret attention in transformers as
a nonparametric kernel regression. We then propose the FourierFormer, a new
class of transformers in which the dot-product kernels are replaced by the
novel generalized Fourier integral kernels. Different from the dot-product
kernels, where we need to choose a good covariance matrix to capture the
dependency of the features of data, the generalized Fourier integral kernels
can automatically capture such dependency and remove the need to tune the
covariance matrix. We theoretically prove that our proposed Fourier integral
kernels can efficiently approximate any key and query distributions. Compared
to the conventional transformers with dot-product attention, FourierFormers
attain better accuracy and reduce the redundancy between attention heads. We
empirically corroborate the advantages of FourierFormers over the baseline
transformers in a variety of practical applications including language modeling
and image classification.