Large-scale pre-trained LMs such as BERT [1]}, GPT [2]} and RoBERTa [3]}, have become the current state-of-the-art approaches for contextual reasoning in CMC models, with leaderboards of both datasets stacked with these models or their variants. The approach is based on the fine-tune BERT-based MRC modeling outlined by bert, in which question and context are packed together (with marker embeddings to distinguish) in an input sequence to BERT that outputs contextualized question-aware embeddings for each input token.
