Instead of using AA T − I 2 F to encourage the diversity for scalar attention matrix as in<cite> Lin et al. (2017)</cite> , we propose the following formula to encourage the diversity for vectorial attention matrices. The penalization term on attention matrices is where λ and µ are hyper-parameters which need to be tuned based on a development set. Intuitively, we try to encourage the diversity of any two different A i under the threshold λ.