We scale the image data into 64×64 as the same as previous works [1]}, [2]}.
Our VQ-VAE is pretrained on a single image in the Pororo training dataset. Each image can be quantized into an 8×8 visual token matrix, where each visual token represents a region of 8×8 pixels. We use one NVIDIA A100 40GB to train this model for 48 hours.
For training transformer, both character token planning and visual token completion models use a 6-layer transformer model with dimension size of 768. We set the head number as 6 and train Sparse Transformers [3]} with local and strided attention across space-time with a dropout probability of 0.1.
We use four NVIDIA A100 40GB to train each model for 12 hours. The implementations of the transformer-based models are extended from the DALLE-python code https://github.com/lucidrains/DALLE-pytorch,
and our entire code-base is implemented in PyTorch https://pytorch.org/.
Please note that all the models in this paper are trained from scratch.
