Even as pre-trained language encoders such as BERT are shared across many tasks, the output layers of question answering and text classification models are significantly different. Span decoders are frequently used for question answering and fixed-class, classification layers for text classification. We show that this distinction is not necessary, and that both can be unified as span extraction. A unified, span-extraction approach leads to superior or comparable performance in multi-task learning, low-data and supplementary supervised pretraining experiments on several text classification and question answering benchmarks.
The use of pre-trained encoders for transfer learning in NLP dates back to @cite_2 @cite_3 but has had a resurgence in the recent past. BERT employs the recently proposed Transformer layers in conjunction with a masked language modeling objective as a pre-trained sentence encoder. Prior to BERT, contextualized word vectors were pre-trained using machine translation data and transferred to text classification and question answering tasks. ELMO improved contextualized word vectors by using a language modeling objective instead of machine translation. ULMFit and GPT showed how traditional, causal language models could be fine-tuned directly for a specific task, and GPT-2 showed that such language models can indirectly learn tasks like machine translation, question answering, and summarization.
@cite_3 @cite_2
2427527485 2117130368
We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0 , a significant improvement over a simple baseline (20 ). However, human performance (86.8 ) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at this https URL We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance.