arxiv:2306.09341 region:us

# Human Preference Dataset v2 (HPD v2)

**Human Preference Dataset v2 (HPD v2)** is a large-scale, cleanly-annotated dataset of human preferences for images generated from text prompts.

For more detailed information, please refer to the paper: [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://arxiv.org/abs/2306.09341). 

We also trained [Human Preference Score v2 (HPSv2)](https://github.com/tgxs002/HPSv2), a preference prediction model, on HPD v2.

## Updates

* [07/29/2023] We released the benchmark and HPD v2 test data. HPD v2 train data will be **released sonn**.

## Data Source

![Overview](assets/overview.png)

The prompts in our dataset are sourced from DiffusionDB and MSCOCO Captions. Prompts from DiffusionDB are first cleaned by ChatGPT to remove biased function words. Human annotators are tasked to rank images generated by different text-to-image generative models from the same prompt. Totally there are about 798k pairwise comparisons of images for over 430k images and 107k prompts, 645k pairs for training split and 153k pairs for test split.

Image sources of HPD v2:
|  Source | # of images 
| :-----: | :-----: |
| CogView2 | 73697 |
| DALL·E 2 | 101869 | 
| GLIDE (mini) | 400 |
| Stable Diffusion v1.4 | 101869 |
| Stable Diffusion v2.0 | 101869 | 
| LAFITE | 400 | 
| VQ-GAN+CLIP | 400 |
| VQ-Diffusion | 400 |
| FuseDream | 400 |
| COCO Captions | 28272 |

# Evaluation prompts

We also provide a set of evaluation prompts (benchmark prompts) that involves testing a model on a total of 3200 prompts, with 800 prompts for each of the following styles: “Animation”, “Concept-art”, “Painting”, and “Photo”. 

In this reposity, We include benchmark images generated by mainstream text-to-image generative model based on benchmark prompts. So far, the following models have been included (being continuously updated):

- ChilloutMix
- CogView2
- DALL·E mini
- DALL·E 2
- Deliberate
- DeepFloyd-XL
- Dreamlike Photoreal 2.0
- Epic Diffusion
- FuseDream
- GLIDE
- LAFITE
- Latent Diffusion
- MajicMix Realistic
- Openjourney
- Realistic Vision
- Stable Diffusion v1.4
- Stable Diffusion v2.0
- SDXL Base 0.9
- SDXL Refiner 0.9
- Versatile Diffusion
- VQ-Diffusion
- VQGAN + CLIP

## Structure

Once unzipped, you should get a folder with the following structure:
```
HPD
---- train/
-------- {image_id}.jpg
---- test/
-------- {image_id}.jpg
---- train.json
---- test.json
---- benchmark/
-------- benchmark_imgs/
------------ {model_id}/
---------------- {image_id}.jpg
-------- drawbench/
------------ {model_id}/
---------------- {image_id}.jpg
-------- anime.json
-------- concept-art.json
-------- paintings.json
-------- photo.json
-------- drawbench.json
```

The annotation file, `train.json`, is organized as:
```
[
    {
        'human_preference': list[int], # 1 for preference
        'prompt': str,
        'file_path': list[str],
        'user_hash': str,
    },
    ...
]
```

The annotation file, `test.json`, is organized as:
```
[
    {
        'prompt': str,
        'image_path': list[str],
        'rank': list[int], # ranking for image at the same index in image_path
    },
    ...
]
```

The benchmark prompts file, ie. `anime.json` is pure prompts. The corresponding image can be found in the folder of the corresponding model by indexing the prompt.