task_categories:visual-question-answering size_categories:1M<n<10M language:en license:cc-by-4.0 arxiv:2307.04087 region:us
Scale up visual instruction tuning to millions by GPT-4.
@article{zhao2023svit,
      title={SVIT: Scaling up Visual Instruction Tuning}, 
      author={Zhao, Bo and Wu, Boya and Huang, Tiejun},
      journal={arXiv preprint arXiv:2307.04087},
      year={2023}
}
---
extra_gated_heading: Acknowledge license to accept the repository
extra_gated_prompt: >
  The Beijing Academy of Artificial Intelligence (hereinafter referred to as "we" or "BAAI") provides you with an open-source dataset (hereinafter referred to as "dataset") through the SVIT HuggingFace repository (https://huggingface.co/datasets/BAAI/SVIT). You can download the dataset you need and use it for purposes such as learning, research, and business, while abiding by the usage rules of each original dataset.

  Before you acquire the open-source dataset (including but not limited to accessing, downloading, copying, distributing, using, or any other handling of the dataset), you should read and understand this "SVIT Open-Source Dataset Usage Notice and Disclaimer" (hereinafter referred to as "this statement"). Once you acquire the open-source dataset, regardless of your method of acquisition, your actions will be regarded as acknowledgment of the full content of this statement.

  1. Ownership and Operation Rights

  You should fully understand that the ownership and operation rights of the SVIT HuggingFace repository (including the current and all previous versions) belong to BAAI. BAAI has the final interpretation and decision rights over this platform/tool and the open-source dataset plan.

  You acknowledge and understand that due to updates and improvements in relevant laws and regulations and the need to fulfill our legal compliance obligations, we reserve the right to update, maintain, or even suspend or permanently terminate the services of this platform/tool from time to time. We will notify you of possible situations mentioned above in a reasonable manner such as through an announcement or email within a reasonable time. You should make corresponding adjustments and arrangements in a timely manner. However, we do not bear any responsibility for any losses caused to you by any of the aforementioned situations.

  2. Claim of Rights to Open-Source Datasets

  For the purpose of facilitating your dataset acquisition and use for learning, research, and business, we have performed necessary steps such as format integration, data cleaning, labeling, categorizing, annotating, and other related processing on the third-party original datasets to form the open-source datasets for this platform/tool's users.

  You understand and acknowledge that we do not claim the proprietary rights of intellectual property to the open-source datasets. Therefore, we have no obligation to actively recognize and protect the potential intellectual property of the open-source datasets. However, this does not mean that we renounce the personal rights to claim credit, publication, modification, and protection of the integrity of the work (if any) of the open-source datasets. The potential intellectual property and corresponding legal rights of the original datasets belong to the original rights holders.

  In addition, providing you with open-source datasets that have been reasonably arranged, processed, and handled does not mean that we acknowledge the authenticity, accuracy, or indisputability of the intellectual property and information content of the original datasets. You should filter and carefully discern the open-source datasets you choose to use. You understand and agree that BAAI does not undertake any obligation or warranty responsibility for any defects or flaws in the original datasets you choose to use.

  3. Usage Restrictions for Open-Source Datasets

  Your use of the dataset must not infringe on our or any third party's legal rights and interests (including but not limited to copyrights, patent rights, trademark rights, and other intellectual property and other rights).

  After obtaining the open-source dataset, you should ensure that your use of the open-source dataset does not exceed the usage rules explicitly stipulated by the rights holders of the original dataset in the form of a public notice or agreement, including the range, purpose, and lawful purposes of the use of the original data. We kindly remind you here that if your use of the open-source dataset exceeds the predetermined range and purpose of the original dataset, you may face the risk of infringing on the legal rights and interests of the rights holders of the original dataset, such as intellectual property, and may bear corresponding legal responsibilities.

  4. Personal Information Protection

  Due to technical limitations and the public welfare nature of the open-source datasets, we cannot guarantee that the open-source datasets do not contain any personal information, and we do not bear any legal responsibility for any personal information that may be involved in the open-source datasets.

  If the open-source dataset involves personal information, we do not bear any legal responsibility for any personal information processing activities you may involve when using the open-source dataset. We kindly remind you here that you should handle personal information in accordance with the provisions of the "Personal Information Protection Law" and other relevant laws and regulations.

  To protect the legal rights and interests of the information subject and to fulfill possible applicable laws and administrative regulations, if you find content that involves or may involve personal information during the use of the open-source dataset, you should immediately stop using the part of the dataset that involves personal information and contact us as indicated in "6. Complaints and Notices."

  5. Information Content Management

  We do not bear any legal responsibility for any illegal and bad information that may be involved in the open-source dataset.

  If you find that the open-source dataset involves or may involve any illegal and bad information during your use, you should immediately stop using the part of the dataset that involves illegal and bad information and contact us in a timely manner as indicated in "6. Complaints and Notices."

  6. Complaints and Notices

  If you believe that the open-source dataset has infringed on your legal rights and interests, you can contact us at 010-50955974, and we will handle your claims and complaints in accordance with the law in a timely manner.

  To handle your claims and complaints, we may need you to provide contact information, infringement proof materials, and identity proof materials. Please note that if you maliciously complain or make false statements, you will bear all legal responsibilities caused thereby (including but not limited to reasonable compensation costs).

  7. Disclaimer

  You understand and agree that due to the nature of the open-source dataset, the dataset may contain data from different sources and contributors, and the authenticity, accuracy, and objectivity of the data may vary, and we cannot make any promises about the availability and reliability of any dataset.

  In any case, we do not bear any legal responsibility for any risks such as personal information infringement, illegal and bad information dissemination, and intellectual property infringement that may exist in the open-source dataset.

  In any case, we do not bear any legal responsibility for any loss (including but not limited to direct loss, indirect loss, and loss of potential benefits) you suffer or is related to the open-source dataset.

  8. Others

  The open-source dataset is in a constant state of development and change. We may update, adjust the range of the open-source dataset we provide, or suspend, pause, or terminate the open-source dataset service due to business development, third-party cooperation, changes in laws and regulations, and other reasons.
extra_gated_fields:
  Name: text
  Affiliation: text
  Country: text
  I agree to accept the license: checkbox
extra_gated_button_content: Acknowledge license
license: cc-by-4.0
task_categories:
  - visual-question-answering
language:
  - en
pretty_name: SVIT
size_categories:
  - 1M<n<10M
---

# Dataset Card for SVIT
Scale up visual instruction tuning to millions by GPT-4.

## Dataset Description
- **Repository:** https://github.com/BAAI-DCAI/Visual-Instruction-Tuning
- **Paper:** https://arxiv.org/pdf/2307.04087.pdf

## Introduction
We Scale up Visual Instruction Tuning (SVIT) and propose a large-scale dataset with 4.2 million informative instruction tuning data, including 1.6M conversation QA pairs, 1.6M complex reasoning QA pairs, 106K detailed descriptions and 1.0M referring QA pairs, by prompting GPT-4 with the abundant manual annotations of image.

The dataset is built based on Visual Genome and MS-COCO. The original images and the annotations from Visual Genome and MS-COCO are in "raw" folder. The instructions and responses generated by GPT-4 are in "data" folder. Details about the dataset can be found in GitHub or the paper.

- GitHub: https://github.com/BAAI-DCAI/Visual-Instruction-Tuning
- Paper: https://arxiv.org/pdf/2307.04087.pdf

## License
The dataset is licensed under a Creative Commons Attribution 4.0 License.
It should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use.
The use of original images and annotations from Visual Genome and MS-COCO should comply with the original licenses.

## Contact us
If you have any comments or questions about the dataset, feel free to create an issue in GitHub: https://github.com/BAAI-DCAI/Visual-Instruction-Tuning/issues.