BERT-based readers are trained in two stages.
In the first stage, the BERT encoder is pre-trained on large amounts of raw text data by predicting masked words conditioned on their context. The learning is self-supervised and does not need any labels. Hugging Facehttps://huggingface.co maintains a rich repository of pre-trained language models (PLMs), including the BERT models and their variants, which are developed for various NLP tasks. However, these PLMs are trained on general-domain text corpora (e.g., Web). If the target domain is dramatically different from general domain, we might consider adapting a general-domain PLM using in-domain data by continual pre-training the PLM. For domains with abundant unlabeled text, such as biomedicine, pre-training domain-specific language models from scratch might also be a good choice [1]}.
In the second stage, the answer span predictor is added on top of the BERT encoder, following the single or pair prediction formulation as illustrated in Figure REF , and both the encoder and the answer predictor are fine-tuned using the MC training data.
