Motivated by [1]}, we observe that stacking multiple attention networks, a.k.a., attention heads, is beneficial since each attention network can specialise on capturing different interactions.
Assume that we use a total of \(C\)  attention networks, each network has its own embedding matrix \(\mathbf {W}^{(c)}\in \mathcal {R}^{F \times K}\)  and weight matrix \(\mathbf {v}^{(c)}\in \mathbb {R}^{2\cdot F\times 1}\)  in the feed forward neural network.
There are multiple ways of combining the attention heads, in our experiments we use average according to Equation REF ,
\(\small \mathbf {A}_t[i,j] = \frac{1}{C}\sum _{c=1}^C \frac{exp(LReLU(\mathbf {v}_{c}^\intercal [\mathbf {W}_{c} \mathbf {x}_t^{(i)} \; || \; \mathbf {W}_{c} \mathbf {x}_t^{(j)} ]))}{\sum _{m \in NB(i)} exp(LReLU(v_{c}^\intercal [\mathbf {W}_{c} \mathbf {x}_t^{(i)} \; || \; \mathbf {W}_{c} \mathbf {x}_t^{(j)} ]))}\) 
