task_categories:visual-question-answering size_categories:n<1K arxiv:2308.12067 region:us
---
task_categories:
- visual-question-answering
size_categories:
- n<1K
---

# InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4
[Lai Wei](https://waltonfuture.github.io/), Zihao Jiang, [Weiran Huang](https://www.weiranhuang.com/), [Lichao Sun](https://lichao-sun.github.io/).

**Shanghai Jiao Tong University, Lehigh University**

[Paper](https://arxiv.org/abs/2308.12067), [Link](https://mp.weixin.qq.com/s/s4Acec71v5oMlFkyhlCL_g), [Code](https://github.com/waltonfuture/InstructionGPT-4)

## Introduction
Multimodal large language models acquire their instruction-following capabilities through a two-stage training process: pre-training on image-text pairs and fine-tuning on supervised vision-language instruction data. Recent studies have shown that large language models can achieve satisfactory results even with a limited amount of high-quality instruction-following data. In this paper, we introduce InstructionGPT-4, which is fine-tuned on a small dataset comprising only 200 examples, amounting to approximately 6% of the instruction-following data used in the alignment dataset for MiniGPT-4. We first propose several metrics to access the quality of multimodal instruction data. Based on these metrics, we present a simple and effective data selector to automatically identify and filter low-quality vision-language data. By employing this method, InstructionGPT-4 outperforms the original MiniGPT-4 on various evaluations (e.g., visual question answering, GPT-4 preference). Overall, our findings demonstrate that less but high-quality instruction tuning data is efficient to enable multimodal large language models to generate better output.

## Usage
You can download our vision-language dataset containing only 200 high-quality examples and replace the original cc_sbu_align dataset used in the fine-tuning stage of MiniGPT-4. The training settings are the same as [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4).

If you're using InstructionGPT-4 in your research or applications, please cite using this BibTeX:
```bibtex
@article{wei2023instructiongpt,
  title={InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4},
  author={Wei, Lai and Jiang, Zihao and Huang, Weiran and Sun, Lichao},
  journal={arXiv preprint arXiv:2308.12067},
  year={2023}
}
```