RNNs and their variants such as LSTM (Hochreiter and Schmidhuber, 1997) have shown success in several natural language processing (NLP) tasks, such as entity extraction (Lample et al., 2016; Ma and Hovy, 2016) , relation extraction <cite>(Vu et al., 2016a</cite>; Miwa and Bansal, 2016; Gupta et al., 2016 Gupta et al., , 2018c , language modeling (Mikolov et al., 2010; Peters et al., 2018) , slot filling (Mesnil et al., 2015; Vu et al., 2016b) , machine translation (Bahdanau et al., 2014) , sentiment analysis (Wang et al., 2016; Tang et al., 2015) , semantic textual similarity (Mueller and Thyagarajan, 2016; Gupta et al., 2018a) and dynamic topic modeling (Gupta et al., 2018d) .