Transformer [1]} is composed of multi-head attention (MHA) and fully connected feed forward network (FFN). The FFN consists of a \(1\times 1\)  convolution with ReLU activation. Layer Normalization (Norm) is usually added in each MHA and FFN. The structure of a Transformer layer is illustrated in Fig.REF .
<FIGURE>