Our model design is motivated by one noticeable defect shared by the VAE-RNN based models in previous works (Bowman et al., 2016; Yang et al., 2017; <cite>Xu and Durrett, 2018</cite>; Dieng et al., 2019) .