where \(\mathrm {norm}(\cdot )\)  denotes the layer normalization [1]}. Note that, for easiness of illustration, the above formulation is based on the single head. In implementation, we adopt an 8-head settings for all attention blocks.
