An attention block measures the pairwise similarity of each query and all keys to assign a weight to each value. Then, the output is computed, which is the weighted sum of the values [1]}. The keys, values, and queries are packed together into matrices \(\mathbf {K}\) , \(\mathbf {V}\) , and \(\mathbf {Q}\) , respectively. The output matrix is then computed as follows
\(\text{Attention}(\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \text{softmax}(\frac{\mathbf {Q}\mathbf {K}^T}{\sqrt{d_k}})\mathbf {V},\) 
