When applying VAEs for text modelling, recurrent neural networks (RNNs) 1 are commonly used as the architecture for both encoder and decoder <cite>(Bowman et al., 2016</cite>; Xu and Durrett, 2018; Dieng et al., 2019) .