Automatic question generation from paragraphs is an important and challenging problem, particularly due to the long context from paragraphs. In this paper, we propose and study two hierarchical models for the task of question generation from paragraphs. Specifically, we propose (a) a novel hierarchical BiLSTM model with selective attention and (b) a novel hierarchical Transformer architecture, both of which learn hierarchical representations of paragraphs. 
 We model a paragraph in terms of its constituent sentences, and a sentence in terms of its constituent words. While the introduction of the attention mechanism benefits the hierarchical BiLSTM model, the hierarchical Transformer, with its inherent attention and positional encoding mechanisms also performs better than flat transformer model.
 We conducted empirical evaluation on the widely used SQuAD and MS MARCO datasets using standard metrics. 
 The results demonstrate the overall effectiveness of the hierarchical models over their flat counterparts. 
 Qualitatively, our hierarchical models are able to generate fluent and relevant questions.
 Question Generation (QG) from text has gained significant popularity in recent years in both academia and industry, owing to its wide applicability in a range of scenarios including conversational agents, automating reading comprehension assessment, and improving question answering systems by generating additional training data. Neural network based methods represent the stateof-the-art for automatic question generation. These models do not require templates or rules, and are able to generate fluent, high-quality questions. Most of the work in question generation takes sentences as input (Du & Cardie, 2018; Kumar et al., 2018; Song et al., 2018; Kumar et al., 2019 ). QG at the paragraph level is much less explored and it has remained a challenging problem. The main challenges in paragraph-level QG stem from the larger context that the model needs to assimilate in order to generate relevant questions of high quality. Existing question generation methods are typically based on recurrent neural networks (RNN), such as bi-directional LSTM. Equipped with different enhancements such as the attention, copy and coverage mechanisms, RNN-based models (Du et al., 2017; Kumar et al., 2018; Song et al., 2018) achieve good results on sentence-level question generation. However, due to their ineffectiveness in dealing with long sequences, paragraph-level question generation remains a challenging problem for these models. Recently, Zhao et al. (2018) proposed a paragraph-level QG model with maxout pointers and a gated self-attention encoder. To the best of our knowledge this is the only model that is designed to support paragraph-level QG and outperforms other models on the SQuAD dataset (Rajpurkar et al., 2016) . One straightforward extension to such a model would be to reflect the structure of a paragraph in the design of the encoder. Our first attempt is indeed a hierarchical BiLSTM-based paragraph encoder ( HPE ), wherein, the hierarchy comprises the word-level encoder that feeds its encoding to the sentence-level encoder. Further, dynamic paragraph-level contextual information in the BiLSTM-HPE is incorporated via both word-and sentence-level selective attention. However, LSTM is based on the recurrent architecture of RNNs, making the model somewhat rigid and less dynamically sensitive to different parts of the given sequence. Also LSTM models are slower to train. In our case, a paragraph is a sequence of sentences and a sentence is a sequence of words. The Transformer (Vaswani et al., 2017 ) is a recently proposed neural architecture designed to address some deficiencies of RNNs. Specifically, the Transformer is based on the (multi-head) attention mechanism, completely discarding recurrence in RNNs. This design choice allows the Transformer to effectively attend to different parts of a given sequence. Also Transformer is relatively much faster to train and test than RNNs. As humans, when reading a paragraph, we look for important sentences first and then important keywords in those sentences to find a concept around which a question can be generated. Taking this inspiration, we give the same power to our model by incorporating word-level and sentence-level selective attention to generate high-quality questions from paragraphs. In this paper, we present and contrast novel approachs to QG at the level of paragraphs. Our main contributions are as follows: • We present two hierarchical models for encoding the paragraph based on its structure. We analyse the effectiveness of these models for the task of automatic question generation from paragraph. • Specifically, we propose a novel hierarchical Transformer architecture. At the lower level, the encoder first encodes words and produces a sentence-level representation. At the higher level, the encoder aggregates the sentence-level representations and learns a paragraphlevel representation. • We also propose a novel hierarchical BiLSTM model with selective attention, which learns to attend to important sentences and words from the paragraph that are relevant to generate meaningful and fluent questions about the encoded answer. • We also present attention mechanisms for dynamically incorporating contextual information in the hierarchical paragraph encoders and experimentally validate their effectiveness. In Table 1 and Table 2 we present automatic evaluation results of all models on SQuAD and MS MARCO datasets respectively. We present human evaluation results in Table 3 and Table 4 respectively. A number of interesting observations can be made from automatic evaluation results in Table 1 and Table 4 : Human evaluation results (column "Score") as well as inter-rater agreement (column "Kappa") for each model on the MS MARCO test set. The scores are between 0-100, 0 being the worst and 100 being the best. Best results for each metric (column) are bolded. The three evaluation criteria are: (1) syntactically correct (Syntax), (2) semantically correct (Semantics), and (3) relevant to the text (Relevance). • Overall, the hierarchical BiLSTM model HierSeq2Seq + AE shows the best performance, achieving best result on BLEU2-BLEU4 metrics on both SQuAD dataset, whereas the hierarchical Transformer model TransSeq2Seq + AE performs best on BLEU1 and ROUGE-L on the SQuAD dataset. • Compared to the flat LSTM and Transformer models, their respective hierarchical counterparts always perform better on both the SQuAD and MS MARCO datasets. • On the MS MARCO dataset, we observe the best consistent performance using the hierarchical BiLSTM models on all automatic evaluation metrics. • On the MS MARCO dataset, the two LSTM-based models outperform the two Transformer-based models. Interestingly, human evaluation results, as tabulated in Table 3 and Table 4 , demonstrate that the hierarchical Transformer model TransSeq2Seq + AE outperforms all the other models on both datasets in both syntactic and semantic correctness. However, the hierarchical BiLSTM model HierSeq2Seq + AE achieves best, and significantly better, relevance scores on both datasets. From the evaluation results, we can see that our proposed hierarchical models demonstrate benefits over their respective flat counterparts in a significant way. Thus, for paragraph-level question generation, the hierarchical representation of paragraphs is a worthy pursuit. Moreover, the Transformer architecture shows great potential over the more traditional RNN models such as BiLSTM as shown in human evaluation. Thus the continued investigation of hierarchical Transformer is a promising research avenue. In the Appendix, in Section B, we present several examples that illustrate the effectiveness of our Hierarchical models. In Section C of the appendix, we present some failure cases of our model, along with plausible explanations. We proposed two hierarchical models for the challenging task of question generation from paragraphs, one of which is based on a hierarchical BiLSTM model and the other is a novel hierarchical Transformer architecture. We perform extensive experimental evaluation on the SQuAD and MS MARCO datasets using standard metrics. Results demonstrate the hierarchical representations to be overall much more effective than their flat counterparts. The hierarchical models for both Transformer and BiLSTM clearly outperforms their flat counterparts on all metrics in almost all cases. Further, our experimental results validate that hierarchical selective attention benefits the hierarchical BiLSTM model. Qualitatively, our hierarchical models also exhibit better capability of generating fluent and relevant questions.
Automatic question generation from paragraph using hierarchical models
{"src": "scitldr-AIC_train", "paper_id": "BJeVXgBKDH"}