The stochastic RNN-based model (Figure REF .(b)) is SVG-LP [1]}. The predictor network is a two layer LSTMs with 256 cells in each layer. The posterior and prior networks are both single layer LSTMs with 256 cells in each layer. Each network has a linear embedding layer and a fully connected output layer. The output dimensionalities of the LSTM networks are \(|g|= 512\) ; \(|\mu _{posterior}| = |\mu _{prior}| = 256\) . The encoder and decoder has the same architecture of the proposed U-net. We replaced the last layer of encoder by a stride-1 \(16\times 16\)  convolution followed by batch normalization and tanh nonlinearity. The encoder outputs a vector with dimensionality \(|h| = 512\) . We added a \(16\times 16\)  deconvolution at the beginning of the decoder to output 512 feature maps with a \(16\times 16\)  size. This model follows “sequence modeling” architecture also named “one to many” architecture used in natural language processing (NLP). To generate a sequence of 6 images (the future), we give the first 6 images (the past) as input to the network. Starting from 6 time step, the generated output is passed as input into the next time step.
