Transformer layers <cite>[11]</cite> have the ability to learn long range relationships for many sequential classification tasks [12] . Multi- The dot product between keys and queries is scaled by the inverse square root of the key dimension. This self-attention operation is done h times in parallel, for the case of h attention heads, with different projection matrices from dinput to d k , d k , and dv. The final output is a concatenation of h vectors each with dimension dv which is in turn linearly projected to the desired output dimension of the self-attention layer. On top of the self-attention component, transformer layers have multiple operations applied on each time step; dropout, residual connection, layer norm, two fully connected layers with a ReLU layer in between, another residual and Layer norm operations.