We thank PlusLab members at ULCA and USC for the helpful discussions, and the anonymous reviewers for useful feedback. This work was supported by DARPA Machine Common Sense (MCS) program under Cooperative Agreement N66001-19-2-4032, a Meta AI SRA, and JSPS/MEXT KAKENHI Grant Numbers JP19H04166 and JP22H05015. Part of the research results was obtained from the commissioned research (No. 225) by National Institute of Information and Communications Technology (NICT), Japan. Hong Chen was financially supported by Chair for Frontier AI Education, The University of Tokyo.

Appendix
More examples
We show another two examples of the results generated by different methods.
In Figure REF , we observe that the image sequence generated by our VP-CSV precisely matches the given story. The characters are likewise compatible with the gold image sequence; however, the characters are unclear in the second image. In Figure REF , compared with baseline model VQ-VAE-LM, VP-CSV generates better `Pororo” and “Crong”. Besides, the results of GAN-based approaches (i.e. VLC and CP-CSV) are exceedingly imprecise, making it almost impossible to distinguish the characters in the image sequence.
<FIGURE><FIGURE>Results on Flintstones dataset.
FlintstonesSV [1]} is another story visualization dataset that is derived by Flintstones dataset [2]}.
To show the generalization of our proposed VP-CSV method, we provide an example on Flintstones dataset in Figure REF . We can observe that compared with VQ-VAE-LM, our proposed method provides higher character quality.
<FIGURE>
Grad-CAM results
In Figure REF , we show some Grad-CAM results on each character. We can see that most of the character regions are highlighted, indicating the high performance of our character region extraction.
<FIGURE>Analysis on Grad-CAM variants.
Since many recent works have extended the original Gran-CAM, we compare the results of different CAM methods, including XGrad-CAM [3]}, Grad-CAM++ [4]}, Eigen-CAM [5]}, Ablation-CAM [6]} and Score-CAM [7]}. In Table REF , we can see that almost all the CAM methods (except Eigen-CAM) can capture the characters' regions. However, in terms of the inference time, Grad-CAM is the fastest as it is the most straightforward method.
Therefore, we use Grad-CAM in our experiment for time-saving character region extraction.
<TABLE><FIGURE>
Limitations on Pororo-SV dataset
In this section, we discuss the limitations of the Pororo-SV dataset. Figure REF  and Figure REF  show bad training samples in the Pororo-SV dataset. Figure REF  shows serious repetition issues (in red) in some of the input stories. Obviously, concatenating these short paragraphs cannot compose a story, thus deviating from purpose of story visualization task.
It will cause our model to learn lousy alignment between the story and the image sequence.
Besides, repetition issues can also happen on the image side. As shown in Figure REF . Due to the frame extraction problem, sometimes the image sequence in the dataset is almost in one scene with no significant changes. This will cause our model to output similar images in the resulted image sequence.

Train-Test discrepancy
In this paper, we propose using VQ-VAE-LM and adapting its Transformer into a two-stage model. We encounter two train-test discrepancy issues: independent training of VQ-VAE and Transformer and the independent training of two stages. Due to the extraordinarily high time-consuming joint training of these models, we place them into our future works.

Implementation details
We scale the image data into 64×64 as the same as previous works [8]}, [1]}.
Our VQ-VAE is pretrained on a single image in the Pororo training dataset. Each image can be quantized into an 8×8 visual token matrix, where each visual token represents a region of 8×8 pixels. We use one NVIDIA A100 40GB to train this model for 48 hours.
For training transformer, both character token planning and visual token completion models use a 6-layer transformer model with dimension size of 768. We set the head number as 6 and train Sparse Transformers [10]} with local and strided attention across space-time with a dropout probability of 0.1.
We use four NVIDIA A100 40GB to train each model for 12 hours. The implementations of the transformer-based models are extended from the DALLE-python code https://github.com/lucidrains/DALLE-pytorch,
and our entire code-base is implemented in PyTorch https://pytorch.org/.
Please note that all the models in this paper are trained from scratch.
Hyperparameters
The results are averaged over three runs until performance convergence is observed.
We tame the image patch size in [4,8,12,16], the transformer dimension in [512, 768, 1024], the number of visual tokens in [64, 128, 256, 512], and choose the hyperparameters with the best score on FID (i.e. image quality).

Annotation instruction
We request AMT for human evaluation. We first launch the qualification round to determine the annotators' qualifications. Each annotator can submit up to 10 HITs ($1.5 for each annotation). We provide detailed instructions to guide annotation. Finally, we send $8 bonus for qualified annotators and invite them for the next rounds. The instruction is shown in Figure REF .
<FIGURE>