The number of parameters added to the network in the above variations is \(d \times N\)  and \(N \times N\)  respectively. When the sequence length is large, these synthetic attention models can be slightly harder to train. Hence, we also included factorized variations, which allow the models to perform competitively in practice. In addition, this form of attention also seems to help prevent overfitting. For details on how to calculate attention the reader is referred to [1]}.
