annotations_creators:crowdsourced language_creators:found size_categories:10K<n<100K source_datasets:original language:en commonsense-reasoning explanation-generation visual-commonsense-reasoning compositionality image-generation visual-question-answering(VQA) question-answering image-captioning arxiv:2303.07274 region:us
---
annotations_creators:
- crowdsourced
language:
- en
language_creators:
- found
paperswithcode_id: whoops
pretty_name: WHOOPS!
size_categories:
- 10K<n<100K
source_datasets:
- original
tags:
- commonsense-reasoning
- explanation-generation
- visual-commonsense-reasoning
- compositionality
- image-generation
- visual-question-answering(VQA)
- question-answering
- image-captioning


task_ids: []
#   dataset files.
extra_gated_prompt: >-
  # By clicking “Access repository“ below, you assert your intention to exclusively use this resource for research, not for commercial chatbot development, and agree to abide by the terms detailed in the [WHOOPS! license](https://whoops-benchmark.github.io/static/pdfs/whoops_license_agreement.txt). You may also view all instances through the [WHOOPS! Explorer](https://huggingface.co/spaces/nlphuji/whoops-explorer-full) and consult the accompanying [WHOOPS! Dataset card](https://huggingface.co/spaces/nlphuji/whoops-explorer-full/blob/main/README.md) prior to acceptance. If you are unsure about your specific case - do not hesitate to reach out: yonatanbitton1@gmail.com.
  By clicking “Access repository” below, you confirm your understanding that for commercial models, this resource is permitted for use as a test set, but not as a training set. Please ensure adherence to the terms detailed in the [WHOOPS! license](https://whoops-benchmark.github.io/static/pdfs/whoops_license_agreement.txt). You may view all instances via the [WHOOPS! Explorer](https://huggingface.co/spaces/nlphuji/whoops-explorer-full) and refer to the [WHOOPS! Dataset card](https://huggingface.co/spaces/nlphuji/whoops-explorer-full/blob/main/README.md) prior to acceptance. If you are unsure about your specific case, don't hesitate to contact: yonatanbitton1@gmail.com.
---


# Dataset Card for WHOOPS!

- [Dataset Description](#dataset-description)
- [Contribute Images to Extend WHOOPS!](#contribute-images-to-extend-whoops)
  - [Languages](#languages)
- [Dataset](#dataset-structure)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
  - [Data Loading](#data-loading)
- [Licensing Information](#licensing-information)
- [Annotations](#annotations)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Citation Information](#citation-information)


## Dataset Description
WHOOPS! is a dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. It contains commonsense-defying image from a wide range of reasons, deviations from expected social norms and everyday knowledge.

The WHOOPS! benchmark includes four tasks:
1. A novel task of explanation-of-violation: generating a detailed explanation for what makes the image weird.
2. Generating a literal caption
3. Distinguishing between detailed and underspecified captions
4. Answering questions that test compositional understanding

The results show that state-of-the-art models such as GPT3 and BLIP2 still lag behind human performance on WHOOPS!.

* Homepage: https://whoops-benchmark.github.io/
* Paper: https://arxiv.org/pdf/2303.07274.pdf
* WHOOPS! Explorer: https://huggingface.co/spaces/nlphuji/whoops-explorer-full
* Normal vs. Wired Explorer: https://huggingface.co/spaces/nlphuji/whoops-explorer-analysis
* Point of Contact: yonatanbitton1@gmail.com

[//]: # (Colab notebook code for WHOOPS evaluation )

## Contribute Images to Extend WHOOPS!
Would you like to add a commonsense-defying image to our database? Please send candidate images to yonatanbitton1@gmail.com. Thanks!

### Languages
English.

## Dataset 
### Data Fields
    image (image) - The weird image.
    designer_explanation (string) - Detailed single-sentence explanation given by the designer, explaining why the image is weird.
    selected_caption (string) - The caption that was selected from the crowed collected captions.
    crowd_captions (list) - Crowd collected captions, depicting whats been seen in the image.
    crowd_explanations (list) - Crowd collected single-sentence explanations, explaining why the image is weird.
    crowd_underspecified_captions (list) - Crowd collected under-specified captions, depicting what is seen in the image, without depicting the commonsense-violation.
    question_answering_pairs (list) - Automatically generated Q-A pairs. FlanT5 XL was used to answer the questions and filter out instances where the BEM metric is above 0.1.
    commonsense_category (string) - The commonsense category the images related to (Full categories list can be found in [paper](https://arxiv.org/pdf/2303.07274.pdf)).
    image_id (string)- The unique id of the image in the dataset
    image_designer (string) - The name of the image designer.

### Data Splits
There is a single TEST split. 
Although primarily intended as a challenging test set, we trained on the WHOOPS! dataset to demonstrate the value of the data and to create a better model. 
We will provide the splits in the future.

### Data Loading
You can load the data as follows (credit to [Winoground](https://huggingface.co/datasets/facebook/winoground)):
```
from datasets import load_dataset
examples = load_dataset('nlphuji/whoops', use_auth_token=<YOUR USER ACCESS TOKEN>)
```
You can get `<YOUR USER ACCESS TOKEN>` by following these steps:
1) log into your Hugging Face account
2) click on your profile picture
3) click "Settings"
4) click "Access Tokens"
5) generate an access token

## Licensing Information
[CC-By 4.0](https://creativecommons.org/licenses/by/4.0/)  
Additional license information: [license_agreement.txt](https://huggingface.co/datasets/nlphuji/whoops/blob/main/license_agreement.txt)  
You may also view all instances through the [WHOOPS! Explorer](https://huggingface.co/spaces/nlphuji/whoops-explorer-full) and consult the accompanying [WHOOPS! Dataset card](https://huggingface.co/spaces/nlphuji/whoops-explorer-full/blob/main/README.md). 

1. **Purpose:** The dataset was primarily designed for use as a test set.
2. **Commercial Use:** Commercially, the dataset may be used as a test set, but it's prohibited to use it as a training set.
3. **Rights on Images:** All rights to the images within the dataset are retained by the WHOOPS! authors.

If you are unsure about your specific case - do not hesitate to reach out: yonatanbitton1@gmail.com.

[//]: # (To evaluate WHOOPS! with a fine-tune BLIP2, we split the images in WHOOPS! into 5 cross- validation splits. For these 5 splits independently, we train supervised models using 60% of the data as training, 20% as validation, and 20% for test.)


## Annotations
We paid designers to create images, and supply explanations for what is making the image wierd.
We paid Amazon Mechanical Turk Workers to supply explanations, captions and under-specified captions for each image in our dataset.

## Considerations for Using the Data
We took measures to filter out potentially harmful or offensive images and texts in WHOOPS!, but it is still possible that some individuals may find certain content objectionable. 
If you come across any instances of harm, please report them to our point of contact. We will review and eliminate any images from the dataset that are deemed harmful.

[//]: # (All images, explanations, captions and under-specified captions were obtained with human annotators.)


### Citation Information
    @article{bitton2023breaking,
      title={Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images},
      author={Bitton-Guetta, Nitzan and Bitton, Yonatan and Hessel, Jack and Schmidt, Ludwig and Elovici, Yuval and Stanovsky, Gabriel and Schwartz, Roy},
      journal={arXiv preprint arXiv:2303.07274},
      year={2023}
    }