Our model design is motivated by one noticeable defect shared by the VAE-RNN based models in previous works <cite>(Bowman et al., 2016</cite>; Yang et al., 2017; Xu and Durrett, 2018; Dieng et al., 2019) .