arxiv:2305.01569 arxiv:2303.14420 arxiv:2304.05977 arxiv:2210.03927 arxiv:2210.08402 region:us
---
dataset_info:
  features:
  - name: are_different
    dtype: bool
  - name: best_image_uid
    dtype: string
  - name: caption
    dtype: string
  - name: created_at
    dtype: timestamp[ns]
  - name: has_label
    dtype: bool
  - name: image_0_uid
    dtype: string
  - name: image_0_url
    dtype: string
  - name: image_1_uid
    dtype: string
  - name: image_1_url
    dtype: string
  - name: jpg_0
    dtype: binary
  - name: jpg_1
    dtype: binary
  - name: label_0
    dtype: float64
  - name: label_1
    dtype: float64
  - name: model_0
    dtype: string
  - name: model_1
    dtype: string
  - name: ranking_id
    dtype: int64
  - name: user_id
    dtype: int64
  - name: num_example_per_prompt
    dtype: int64
  - name: __index_level_0__
    dtype: int64
  splits:
  - name: train
    num_bytes: 193273338802
    num_examples: 583747
  - name: validation
    num_bytes: 5638295249
    num_examples: 17439
  - name: test
    num_bytes: 4621428929
    num_examples: 14073
  - name: validation_unique
    num_bytes: 178723392
    num_examples: 500
  - name: test_unique
    num_bytes: 178099641
    num_examples: 500
  download_size: 202289408791
  dataset_size: 203889886013
---

# Dataset Card for Pick-a-Pic (v1)

## Dataset Description

- **Homepage: The web app can be found at [pickapic.io](https://pickapic.io/)**
- **Repository: The repository of [PickScore](https://github.com/yuvalkirstain/PickScore)**
- **Paper: [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://arxiv.org/abs/2305.01569).**
- **Leaderboard: TODO **
- **Point of Contact: TODO **

### Dataset Summary

The Pick-a-Pic dataset was collected with the [Pick-a-Pic web app](https://pickapic.io/) and contains over half-a-million examples of human preferences over model-generated images.
This dataset with URLs instead of the actual images (which makes it much smaller in size) can be found [here](https://huggingface.co/datasets/yuvalkirstain/pickapic_v1_no_images).

See the corresponding paper [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://arxiv.org/abs/2305.01569) for more details.

If you want to download this dataset with URLs instead of images to save space, please see [this version of the dataset](https://huggingface.co/datasets/yuvalkirstain/pickapic_v1_no_images).

### Supported Tasks and Leaderboards

Task: Select preferred image in test-set.
| **Models** | **Test-Set Accuracy (%)** |
| --- | --- |
| [PickScore](https://arxiv.org/abs/2305.01569) | 70.2% |
| Human Expert Baseline | 68.0% |
| [HPS](https://arxiv.org/abs/2303.14420) | 66.7% |
| [ImageReward](https://arxiv.org/abs/2304.05977) | 61.1% |
| [CLIP-H](https://arxiv.org/abs/2210.03927) | 60.8% |
| [Aesthetics](https://arxiv.org/abs/2210.08402) | 56.8% |

### Data Splits

The dataset has three main splits: train, validation, validation_unique (with one example per prompt), test, and test_unique.


### Citation Information
If you find this work useful, please cite:
```bibtex
@inproceedings{Kirstain2023PickaPicAO,
  title={Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation},
  author={Yuval Kirstain and Adam Polyak and Uriel Singer and Shahbuland Matiana and Joe Penna and Omer Levy},
  year={2023}
}
```

### LICENSE
MIT License

Copyright (c) 2021

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
