[1]}'s Synthesizer Transformer is able to learn attention weights synthetically, without token-token interaction. This increases the speed of the Transformer by up to 60%. Synthesizer Transformers can do this by removing the notion of query-key-values in the self-attention calculation and instead directly synthesizing the attention matrix. This is done using input \(X_{h,l} \in \mathbb {R}^{N \times d}\)  , where \(h\)  is the number of heads, \(l\)  is the sequence length and \(d\)  is the dimensionality of the model. This eliminates the need to calculate the dot product attention as described in the previous subsection. In their original paper, [1]} propose several synthetic attention variants, in this work, we implemented some of the best performing variants: dense, random, both of their factorized version, as well as a combination of dense and random with the Vanilla Transformer attention.
<FIGURE>