When applying VAEs for text modelling, recurrent neural networks (RNNs) 1 are commonly used as the architecture for both encoder and decoder (Bowman et al., 2016; <cite>Xu and Durrett, 2018</cite>; Dieng et al., 2019) . While such a VAE-RNN based architecture allows encoding and generating sentences (in the decoding phase) with variablelength effectively, it is also vulnerable to an issue known as latent variable collapse (or KL loss vanishing) , where the posterior collapses to the prior and the model will ignore the latent codes in generative tasks.