The different attention mechanisms: Attention mechanisms have become an integral part of sequence modeling tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences. Specifically, we experiment with two attention mechanisms: (i) vanilla self-cross attention and (ii) gated cross attention. The Vanilla self-cross attention is the basic attention mechanism used in traditional transformer models [1]}. The self-attention module relates different positions of an input sequence in order to compute a representation of the input sequence. This module is present in both encoder and decoder layers. The cross attention module relates different positions of an input sequence to the output sequence. This module connects the encoder and decoder components. We term AutoCoder variant that uses the standard self-cross attention module as Vanilla-AutoCoder (VAC).
\(f_{SA}=f_{dot}(Q_e,K_e,V_e)=softmax\left( \frac{Q_eK_e^T}{\sqrt{d}} \right) V_e\) 
\(f_{CA}=f_{dot}(Q_e,K_d,V_d)=softmax\left( \frac{Q_eK_d^T}{\sqrt{d}} \right) V_d\) 
