In the original work, the authors proposed sinusoidal functions for pre-calculating PE vectors for the input dataset [1]}. The PE vectors do not include any learnable parameters and are added directly to the word embedding input vectors. Figure REF  shows the formulation of PE vectors with eqs. REF  and ,
\(\text{PE}_{(pos, 2i)} &= \mathbf {p}_i &&= \sin \left(\frac{pos}{10000^\frac{2i}{d}}\right),  \\\text{PE}_{(pos, 2i+1)} &= \mathbf {p}_{i+1} &&= \cos \left(\frac{pos}{10000^\frac{2i}{d}}\right), \) 
