Our model design is motivated by one noticeable defect shared by the VAE-RNN based models in previous works (Bowman et al., 2016; <cite>Yang et al., 2017</cite>; Xu and Durrett, 2018; Dieng et al., 2019) .