annotations_creators:crowdsourced language_creators:found multilinguality:monolingual size_categories:1K<n<10K source_datasets:original language:en license:cc-by-4.0 image-captioning text-image-matching arxiv:2305.10400 region:us
---
annotations_creators:
- crowdsourced
language:
- en
language_creators:
- found
license:
- cc-by-4.0
multilinguality:
- monolingual
paperswithcode_id: seetrue
pretty_name: SeeTRUE
size_categories:
- 1K<n<10K
source_datasets:
- original
tags:
- image-captioning
- text-image-matching
task_ids: []
extra_gated_prompt: "By clicking on “Access repository” below, you also agree that you are using it solely for research purposes, and that SeeTRUE should be used as a *TEST SET*, not as a training set, and especially not to train commercial chatbots. Do not hessitate to contact yonatanbitton@google.com if you have questions about this license."
---
# Dataset Card for SeeTRUE

- [Dataset Description](#dataset-description)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## Dataset Description
The SeeTRUE dataset is a diverse benchmark for meta-evaluation of image-text alignment methods, covering the 4-way combinations of real and synthetic text-and-image pairs. It addresses limitations in current benchmarks, which mainly focus on natural images and often lack challenging negative captions. SeeTRUE allows to better assess the generalization abilities of text-image alignment models across various tasks.  
We will add more datasets from SeeTRUE (e.g., COCO-Con and PickaPic-Con) upon data release.   
Paper: https://arxiv.org/abs/2305.10400  
Website: https://wysiwyr-itm.github.io/  

### Languages
The dataset supports English language.

## Dataset Structure
### Data Fields
- image: The name of the image file.
- text: The text description that matches with the image.
- label: The binary label. 1 if the text matches with the image, 0 otherwise.
- original_dataset_id: The ID of the dataset where the row originates from.
- dataset_source: The source of the dataset.

### Data Splits
SeeTRUE contains a single split: TEST, and should not be used for training. 

## Dataset Creation
The dataset has been created by sourcing and matching images and text from multiple datasets. More information in the paper: https://arxiv.org/abs/2305.10400.

### Licensing Information
The dataset is under the CC-By 4.0 license.

### Citation Information
@article{yarom2023you,
  title={What You See is What You Read? Improving Text-Image Alignment Evaluation},
  author={Yarom, Michal and Bitton, Yonatan and Changpinyo, Soravit and Aharoni, Roee and Herzig, Jonathan and Lang, Oran and Ofek, Eran and Szpektor, Idan},
  journal={arXiv preprint arXiv:2305.10400},
  year={2023}
}