The attention mechanism [1]} has been widely used in many sequence modeling tasks.
Its dot-product variant is the key building block for the
state-of-the-art transformer architectures [2]}.
Let \(\lbrace {\mathbf {q}}_t\rbrace _{t=1}^{N}\)  denote
a sequence of \(N\)  query vectors,
that attend to sequences of \(M\)  key and value vectors.
At each timestep, the attention
linearly combines the values weighted by the outputs of a softmax:
\(\begin{split}\operatorname{attn}\left({\mathbf {q}}_t,\lbrace {\mathbf {k}}_i\rbrace ,\lbrace {\mathbf {v}}_i\rbrace \right)=\sum _i\frac{\exp \left({\mathbf {q}}_t\cdot {\mathbf {k}}_i /\tau \right)}{\sum _j\exp \left({\mathbf {q}}_t\cdot {\mathbf {k}}_j /\tau \right)}{\mathbf {v}}_i^\top .\end{split}\) 
