To enable the three tasks, we show how to modify the attention masking mechanism in Transformer [1]}.
Transformer consists of \(L\)  identical layers, each of which is composed of two sub-layers: multi-head self-attention and position-wise feed-forward network.
The \(l\) -th layer encodes the previous layer's output \(\mathbf {S}_{l - 1}\)  into \(\mathbf {S}_l\) , where \(l \in [1, L]\) .
In the multi-head self-attention sub-layer, the computation of each attention head is also identical, and among the \(H\)  heads of the \(l\) -th layer, the \(h\) -th head \(\mathbf {A}_{l, h}\)  is computed as follows:
\(\begin{aligned}\mathbf {A}_{l, h} & = \text{softmax} (\frac{\mathbf {Q}_{l, h} \mathbf {K}_{l, h}^\top }{\sqrt{d}} + \mathbf {M}) \mathbf {V}_{l, h} \\\mathbf {Q}_{l, h} & = \mathbf {S}_{l - 1} \mathbf {W}_{l, h}^Q, \mathbf {K}_{l, h} = \mathbf {S}_{l - 1} \mathbf {W}_{l, h}^K, \\\mathbf {V}_{l, h} & = \mathbf {S}_{l - 1} \mathbf {W}_{l, h}^V \\\mathbf {M} & ={\left\lbrace \begin{array}{ll}0, & \text{Allow to attend} \\- \infty , & \text{Prevent from attending}\end{array}\right.}\end{aligned}\) 
