where \(W^q_j, W^k_j, W^v_j \in \mathbb {R}^{D \times \gamma }\)  are weight matrices for computing the (query, key, value) triplet given the input \(\mathbf {x}\) . This operation is repeated for each attention head, and the outputs are combined (with concatenation followed by one linear layer with weights \(W^b \in \mathbb {R}^{D \times D}\) ), producing \(\mathbf {a}\in \mathbb {R}^{T \times D}\) . Next, the feed-forward layer takes this intermediate output and computes \(\mathbf {o}\in \mathbb {R}^{T \times D}\)  using a two-layer fully-connected network with weights \(W^c \in \mathbb {R}^{D \times E}\)  and \(W^d \in \mathbb {R}^{E \times D}\) . The output of each sub-layer is computed using a residual function followed by layer normalization [1]}, i.e., \(\mbox{LayerNorm}(x + \mbox{Sublayer}(x))\) .
In this work, we set the number of layers \(L=6\) , the number of attention heads \(A=12\) , the feature dimension \(D=768\)  and the intermediate dimension \(E=3072\) . For simplicity, we use this design for all layers across all three Transformers in our model.
