annotations_creators:crowdsourced language_creators:found size_categories:10K<n<100K source_datasets:original language:en license:cc-by-4.0 vision-and-language instruction-following human-chatbot-interaction image-instruction-pairs multi-modal task-performance arxiv:2308.06595 region:us
---
annotations_creators:
- crowdsourced
language:
- en
language_creators:
- found
paperswithcode_id: visit-bench
pretty_name: VisIT-Bench
size_categories:
- 10K<n<100K
source_datasets:
- original
tags:
- vision-and-language
- instruction-following
- human-chatbot-interaction
- image-instruction-pairs
- multi-modal
- task-performance
task_ids: []
extra_gated_prompt: >-
  By clicking “Access repository” below, you assert your intention to
  exclusively use this resource for research, not for commercial chatbot
  development, and agree to abide by the terms detailed in the [VisIT-Bench
  license](https://visit-bench.github.io/static/pdfs/visit_bench_license_agreement.txt).
  You may also view all instances through the [VisIT-Bench
  Explorer](https://huggingface.co/spaces/mlfoundations/visit-bench-explorer-full)
  and consult the accompanying [VisIT-Bench Dataset
  card](https://huggingface.co/spaces/mlfoundations/visit-bench-explorer-full/blob/main/README.md)
  prior to acceptance. If you are unsure about your specific case - do not
  hesitate to reach out: visit-bench-support@gmail.com.
license: cc-by-4.0
---

# Dataset Card for VisIT-Bench
- [Dataset Description](#dataset-description)
- [Dataset Structure](#dataset-structure)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
  - [Data Loading](#data-loading)
- [Licensing Information](#licensing-information)
- [Annotations](#annotations)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Citation Information](#citation-information)

## Dataset Description
VisIT-Bench is a dataset and benchmark for vision-and-language instruction following. The dataset is comprised of image-instruction pairs and corresponding example outputs, spanning a wide range of tasks, from simple object recognition to complex reasoning tasks. The dataset provides a holistic view of chatbot capabilities.

The results show that state-of-the-art models such as GPT-4 and BLIP2 have a high success rate, but there is room for improvement.

Homepage: https://visit-bench.github.io/
Paper: https://arxiv.org/abs/2308.06595
GitHub: http://github.com/mlfoundations/Visit-Bench
Point of Contact: yonatanbitton1@gmail.com, hbansal@ucla.edu

## Dataset Structure
### Data Fields
    instruction_category (string) - The category of the instruction
    image_url (string) - The URL of the image in the instruction
    image (image) - The image in the instruction
    visual (string) - The visual details in the instruction
    instruction (string) - The instruction itself
    reference_output (string) - The reference output for the given instruction
    human_ratings_gpt4_correct (boolean) - Human ratings indicating if GPT-4 correctly followed the instruction
    human_ratings_problem_in_caption (boolean) - Human ratings indicating if there is a problem in the caption
    human_ratings_problem_in_gpt4 (boolean) - Human ratings indicating if there is a problem in GPT-4's response
    public_images_metadata (dictionary) - Metadata about the image

### Data Splits
The dataset currently has a single TEST split. Further splits will be provided in the future.

### Data Loading
You can load the data as follows (credit to [Hugging Face Datasets](https://huggingface.co/datasets)):
```
from datasets import load_dataset
examples = load_dataset('mlfoundations/visit-bench', use_auth_token=<YOUR USER ACCESS TOKEN>)
```
You can get `<YOUR USER ACCESS TOKEN>` by following these steps:
1) log into your Hugging Face account
2) click on your profile picture
3) click "Settings"
4) click "Access Tokens
5) generate a new token and use that in the `use_auth_token` field

## Licensing Information
The new contributions of our dataset (e.g., the instructions, reference outputs, model ranking annotations, etc.) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). 
All images used are publically licensed. Please refer to the public license attached to each individual image in the "public_images_metadata" field in the dataset sheets.
Alongside this license, the following conditions apply:
1. **Purpose:** The dataset was primarily designed for use as a test set.
2. **Commercial Use:** Commercially, the dataset may be used as a test set, but it's prohibited to use it as a training set.
By accessing or using this dataset, you acknowledge and agree to abide by these terms in conjunction with the CC BY 4.0 license.

## Annotations
The dataset is annotated using crowd workers on Amazon Mechanical Turk. Workers followed the steps detailed in the paper to generate the annotations. The instructions, reference outputs, and model ranking annotations were generated through this process.

## Considerations for Using the Data
Social Impact of Dataset: The dataset is aimed to facilitate research on AI models' ability to understand and follow instructions given in natural language and paired with visual inputs. Such research could contribute to the development of more interactive, capable, and intelligent AI systems. It could also illuminate areas where current AI technology falls short, informing future research directions.

Data Limitations: The dataset may not cover all possible types of instructions, particularly those requiring complex reasoning or advanced knowledge. The dataset was also created using crowd workers, and thus, may contain mistakes or inconsistencies.

Privacy: The images used in this dataset are publicly available. However, the exact source of the images is not disclosed in the dataset, protecting the privacy of the image creators to some extent. The workers who generated the instructions and annotations were also anonymized.

Curation Rationale: The dataset was curated to provide a broad range of instruction types and difficulty levels. The creators selected a mix of easy, medium, and hard instructions to challenge current AI capabilities.

## Citation Information
@misc{bitton2023visitbench,
      title={VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use}, 
      author={Yonatan Bitton and Hritik Bansal and Jack Hessel and Rulin Shao and Wanrong Zhu and Anas Awadalla and Josh Gardner and Rohan Taori and Ludwig Schimdt},
      year={2023},
      eprint={2308.06595},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
