We employ the relative sinusoidal positional encoding scheme which is an important technique from Transformer-XL[1]}. The relative positional encoding allows the self-attention module to generalize better on different input lengths so that the resulting encoder is more robust to the variance of the utterance length.
<FIGURE><FIGURE>