As shown in Figure REF , an encoder block consists of a multi-head self-attention layer and a feed forward layer connected back-to-back with residual connections and normalization layers. Residual connections are a commonly used technique for training deep neural networks and help in training stabilization and learning [1]}. The layer normalization is also commonly used in neural networks for processing sequential data and helps in faster convergence of the training [2]}. The feed forward layer comprises of two linear layers with a ReLU activation function [3]}. The output of an encoder block is used as an input for the next encoder block. The input to the first encoder block consists of the sum of word embeddings and positional encoding (PE) vectors.
