The Transformer is a popular neural network designed to learn language models, e.g., dependencies between words, in a context. Transformer models have recently shown to have remarkable impact on AS2 when used as ranker [1]}, [2]}, [3]}.
Besides architectural definitions, Transformer models take advantage for extensive pre-training the language model (i.e., the q/a pair representation) on large-scale corpora by using the masked language model and next sentence prediction [4]}.
In those works, the question answer candidate pairs are codified as a joint sequence of tokens with specialized delimiters and separators, i.e.,
[CLS] \(q^1\dots q^n\)  [SEP] \(s^1\dots s^m\)  [EOS], where \(x^j\)  defines the \(j\) -th token of the sequence \(x\) .
[CLS], [SEP], and [EOS] are special tokens used to mark the beginning of the sequence, the separation between question and candidate answer tokens, and the end of the text.
Several Transformer blocks are applied and then the representation associated with [CLS] is used in a linear fully-connected layer to to compute the final score associated with the question/answer pair.
The same concepts can be applied to RoBERTa or other pre-trained Transformer models.
