task_categories:image-to-text task_categories:zero-shot-classification size_categories:100M<n<1B license:cc-by-4.0 arxiv:2307.10350 region:us
---
license: cc-by-4.0
task_categories:
- image-to-text
- zero-shot-classification
size_categories:
- 100M<n<1B
---
# Dataset Card for DataComp_medium_pool_BLIP2_captions

## Dataset Description

- **Paper: https://arxiv.org/abs/2307.10350**
- **Leaderboard: https://www.datacomp.ai/leaderboard.html** 
- **Point of Contact: Thao Nguyen (thaottn@cs.washington.edu)** 

### Dataset Summary



### Supported Tasks and Leaderboards

We have used this dataset for pre-training CLIP models and found that it rivals or outperforms models trained on raw web captions on average across the 38 evaluation tasks proposed by DataComp.
Refer to the DataComp leaderboard (https://www.datacomp.ai/leaderboard.html) for the top baselines uncovered in our work.

### Languages

Primarily English.

## Dataset Structure

### Data Instances

Each instance maps a unique image identifier from DataComp to the corresponding BLIP2 caption generated with different softmax temperatures (see the filename for the temperature value used).

### Data Fields

uid: SHA256 hash of image, provided as metadata by the DataComp team.

blip2-cap: corresponding caption generated by BLIP2.

### Data Splits

Data was not split. The dataset is intended for pre-training multimodal models.

## Dataset Creation

### Curation Rationale

Web-crawled image-text data can contain a lot of noise, i.e. the caption may not reflect the content of the respective image. Filtering out noisy web data, however, can hurt the diversity of the training set.
To address both of these issues, we use image captioning models to increase the number of useful training samples from the initial pool, by ensuring the captions are more relevant to the images. 
Our work systematically explores the effectiveness of using these synthetic captions to replace or complement the raw text data, in the context of CLIP pre-training.

### Source Data

#### Initial Data Collection and Normalization

The original 128M image-text pairs were collected by the DataComp team from Common Crawl. Minimal filtering was performed on the initial data pool (face blurring, NSFW removal, train-test deduplication).
We then replaced the original web-crawled captions with synthetic captions generated by BLIP2.

#### Who are the source language producers?

Common Crawl is the source for images. BLIP2 is the source of the text data.

### Annotations

#### Annotation process

The dataset was built in a fully automated process: captions are generated by the BLIP2 captioning model.

#### Who are the annotators?

No human annotators are involved.

### Personal and Sensitive Information

The images, which we inherit from the DataComp benchmark, already underwent face detection and face blurring. While the DataComp team made an attempt to remove NSFW instances, it is possible that such content may still exist (to a small degree) in this dataset. 
Due to the large scale nature of this dataset, the content has not been manually verified to be completely safe. Therefore, it is strongly recommended that this dataset be used only for research purposes.

## Considerations for Using the Data

### Social Impact of Dataset

The publication contains some preliminary analyses of the fairness implication of training on this dataset, when evaluating on Fairface.

### Discussion of Biases

Refer to the publication for more details.

### Other Known Limitations

Refer to the publication for more details.

## Additional Information

### Citation Information

```bibtex
@article{nguyen2023improving,
  title={Improving Multimodal Datasets with Image Captioning},
  author={Nguyen, Thao and Gadre, Samir Yitzhak and Ilharco, Gabriel and Oh, Sewoong and Schmidt, Ludwig},
  journal={arXiv preprint arXiv:2307.10350},
  year={2023}
}
```