task_categories:text-generation task_categories:visual-question-answering language:en license:cc-by-nc-4.0 llava llavar arxiv:2306.17107 region:us
---
license: cc-by-nc-4.0
task_categories:
- text-generation
- visual-question-answering
language:
- en
tags:
- llava
- llavar
---

# LLaVAR Data: Enhanced Visual Instruction Data with Text-Rich Images

More info at [LLaVAR project page](https://llavar.github.io/), [Github repo](https://github.com/SALT-NLP/LLaVAR), and [paper](https://arxiv.org/abs/2306.17107).

## Training Data

Based on the LAION dataset, we collect 422K pretraining data based on OCR results. For finetuning data, we collect 16K high-quality instruction-following data by interacting with langauge-only GPT-4. Note that we also release a larger and more diverse finetuning dataset below (20K), which contains the 16K we used for the paper. The instruction files below contain the original LLaVA instructions. You can directly use them after merging the images into your LLaVA image folders. If you want to use them independently, you can remove the items contained in the original chat.json and llava_instruct_150k.json from LLaVA.

[Pretraining images](./pretrain.zip)

[Pretraining instructions](./chat_llavar.json)

[Finetuning images](./finetune.zip)

[Finetuning instructions - 16K](./llava_instruct_150k_llavar_16k.json)

[Finetuning instructions - 20K](./llava_instruct_150k_llavar_20k.json)

## Evaluation Data
We collect 50 instruction-following data on 50 text-rich images from LAION. You can use it for GPT-4-based instruction-following evaluation.


[Images](./REval.zip)

[GPT-4 Evaluation Contexts](./caps_laion_50_val.jsonl)

[GPT-4 Evaluation Rules](./rule_read_v3.json)

[Questions](./qa50_questions.jsonl)

[GPT-4 Answers](./qa50_gpt4_answer.jsonl)