For multi-headed attention layers, we follow [1]}, as formulated in Eq. REF . All of the W matrices are learned during the training procedure. Since it is a self-attention layer, query Q is the same as key K and value V. After each attention layer, layer normalization and the residual connection are implemented.
\(\begin{aligned}&\textrm {MultiHead}(Q,K,V) = \textrm {Concat}(\textrm {Head}_1, ..., \textrm {Head}_h)W^O \\&\textrm {Head}_i = \textrm {Attention}(QW_i^Q, KW_i^K, VW_i^V) \\&\textrm {Attention}(Q,K,V) = \sigma \Big (\frac{QK^T}{\sqrt{d}}V\Big )\end{aligned}\) 
<TABLE>