
where \(\max (0,*)\)  represents the ReLU activation function; \(\text{W}_\text{f} \in \mathbb {R}^{d \times 4d}\)  and \(\text{W}_\text{ff} \in \mathbb {R}^{4d \times d}\)  denote learnable matrices for linear transformation; \(\text{b}_\text{f}\)  and \(\text{b}_\text{ff}\)  represent the bias terms. It is worth noting that both the MHA and FFN are followed by an operation sequence of dropout [1]}, residual connection [2]}, and layer normalization [3]}.
