Early extractive readers commonly use sequence models (such as Recurrent Neural Networks) with attention for their encoders. After the Transformer [1]} is proposed and widely used to develop pre-trained language models (such as BERT [2]}), the extractive readers that use pre-trained Transformer models as encoders become state of the art.
In what follows, we use two examples to illustrate how these extractive readers are implemented.
The first is the Bi-Directional Attention Flow (BiDAF) model [3]}, which is one of the most popular baseline readers that use sequence models as encoders.
The second is the state-of-the-art extractive reader based on BERT [2]}.
