The architecture of the transformer encoder is very similar to the standard encoder proposed by [1]} where each layer consists of a layer normalization module, a multi-headed self-attention module, and a feed forward network (FFN) module along with residual connections. The transformer encoder ingests a sequence of elements consisting of: 1) the output from the MLP encoder, 2) the output from CNN encoder (after being flattened to match the dimension of output of MLP encoder) and 3) positional encoding.
