Another approach is to analyze the expressivity of attention networks. One work interprets multi-head self-attention as a Bayesian inference, and provides tools to decide how many heads to use, and how to enforce distinct representations in different heads [1]}. Other analysis has demonstrated that sparse transformers can universally approximate any function [2]}, that multi-head self-attention is at least as expressive as a convolutional layer [3]}, and that dot-product self-attention is not Lipschitz continuous [4]}.
