Reverse rendering from vision to text.Vision-language alignment data focuses on accurately grounding visual input to an LLM (usually via a vision encoder). Web-scraped image-caption pairs have been the most popular MM alignment data in the past few years since CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021). However, web-scraped image-text pairs are usually noisy and only have coarse-grained correspondence, insufficient for grounding details of images in language. In domains such as documents, screens, figures, and diagrams, such fine-grained alignment can most conveniently be obtained from data synthesis pipelines built with image rendering engines. Pix2Struct (Lee et al., 2023) uses web servers to render HTML code into website screenshots, and the training task is to dereder a masked screenshot to the full HTML code. MatCha (Liu et al., 2023b) and DePlot (Liu et al., 2023a) render tabular data into charts with Python plotting libraries and pretrain a foundation model by giving the rendered image and producing the code and/or the tabular data. Si et al. (2024) and Laurencon et al. (2024) train on synthetically generated HTML and CSS files for the task of converting webpage screenshots into code implementation. The models finetuned on the synthetic data can generalize reasonably well on realistic data scraped from the Internet. Borkman et al. (2021) propose to use physics engines or game engines (e.g., Unity) as the synthetic data generator to help computer vision research.