In this paper, we will focus mainly on self-attention, which takes as an input value matrix \(V\)  with dimensionality \(n \times d\)  and for each of \(n\)  vectors outputs weighted average of the whole vector sequence. There are two types of attention mechanism, namely soft attention and hard attention [1]}. Following [2]}, the former assigns the weights for each value vector using this formula:
\(\text{Soft Self-Attention}(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d}}) V\) 
