@inproceedings{
zhang2024pretrained,
title={Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation},
author={Yuhui Zhang and Brandon McKinzie and Zhe Gan and Vaishaal Shankar and Alexander Toshev},
booktitle={I Can't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models},
year={2024},
url={https://openreview.net/forum?id=PxiuaUKf8y}
}
Recent advances in image tokenizers, such as VQ-VAE, have enabled text-to-image generation using auto-regressive methods, similar to language modeling. However, these methods have yet to leverage pre-trained language models, despite their adaptability to various downstream tasks. In this work, we explore this gap by adapting a pre-trained language model for auto-regressive text-to-image generation, and find that pre-trained language models offer limited help. We provide a two-fold explanation by analyzing tokens from each modality. First, we demonstrate that image tokens possess significantly different semantics compared to text tokens, rendering pre-trained language models no more effective in modeling them than randomly initialized ones. Second, the text tokens in the image-text datasets are too simple compared to normal language model pre-training data, which causes the catastrophic degradation of language models' capability.
Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation
Yuhui Zhang Brandon McKinzie Zhe Gan Vaishaal Shankar Alexander Toshev