Encoding layer: Given an input sentences consist of n words \(w_1,w_2 ... w_n\) .
The encoding layer employs a pre-trained language model based on Transformers encoder [1]} like RoBERTa [2]}, Electra [3]}.
Because RoBERTa, Electra segments the input sentence to sub-words which enables the encoding of rare words with appropriate sub-words.
We re-combined sub-word representation after encoding layer into word representation using sum operation. The \(i^{th}\)  word \(w_i\)  is represented by vector \(e_i\)  as follows:
\(e_i = PretrainedLM(w_{1:n},i)\) 
