Multi-modality instruction following.Downstream applications of multimodal LLMs require reasoning and instruction following capabilities. Such data are usually long-form question response pairs and are expensive for humans to create. LLaVA (Liu et al., 2024b) uses existing image captions to prompt GPT-4 (in text-only mode) for writing diverse and long-form prompt-answer pairs. During multimodal LLM training, images and prompts are used as input while the captions and bounding box information can be hidden. Besides image captions, other sources of image attribute information such as object bounding box (Zhao et al., 2023), OCR (Zhang et al., 2023d) and deredered charts (Carbune et al., 2024; Masry et al., 2023) can all fit into such as image attributes + text LLM rewriting synthetic data pipeline.