task_categories:text-to-image task_categories:image-to-text task_ids:image-captioning multilinguality:monolingual size_categories:10K<n<100K language:en license:other region:us
ULTIMA Dataset - Uma Musume Labeled Text-Image Multimodal Alignment Dataset
@misc{ULTIMA,
  author       = {Oh Giyeong (BootsofLagrangian), Kang Dohoon (Haken)},
  title        = {ULTIMA - Uma Musume Labeled Text-Image Multimodal Alignment Dataset},
  howpublished = {\\url{https://huggingface.co/datasets/UmaDiffusion/ULTIMA}},
  month        = {July},
  year         = {2023}
}
---
license: other
language:
- en
multilinguality:
  - monolingual
pretty_name: Uma Musume Labeled Text-Image Multimodal Alignment Dataset
size_categories:
- 10K<n<100K
task_categories:
  - text-to-image
  - image-to-text
task_ids:
  - image-captioning
extra_gated_prompt: "You agree to use this dataset for non-commercial ONLY and NOT VIOLATE the guidelines for secondary creation of Uma Musume Pretty Derby."
extra_gated_fields:
 I agree to use this dataset for non-commercial ONLY and to NOT VIOLATE the guidelines for secondary creation of Uma Musume Pretty Derby from Cygames, Inc: checkbox
 
---
--- 
# About **ULTIMA**

ULTIMA Dataset is **U**ma Musume **L**abeled **T**ext-**I**mage **M**ultimodal **A**lignment Dataset.   

ULTIMA is *a supervised dataset for fine-tuning* of characters in Uma Musume: Pretty Derby.  

It contains **~14K** text-image pairs.  

We ***manually*** processed the entire data. This is an essential fact even though it is assisted by a machine.  

What we did is on [Data Preprocessing.md](https://huggingface.co/datasets/UmaDiffusion/ULTIMA/blob/main/Data%20Preprocessing.md).  
Statistics about datset and abbreviations of Uma Musume are in [statistics.md](https://huggingface.co/datasets/UmaDiffusion/ULTIMA/blob/main/statistics.md).  
Pruned tag-clothes pairs are in [prompts.md](https://huggingface.co/datasets/UmaDiffusion/ULTIMA/blob/main/prompts.md)


## Dataset Structure

We use a modularized file structure to distribute ULTIMA. The 14,460 images in ULTIMA are split into 73 folders, where each folder contains 200 images and a JSON file that these 200 images to their text and information.  

```bash
# ULTIMA
./
├──data
│  ├──part-00000 
│  │  ├──01_agt_00000.png
│  │  ├──01_agt_00001.png
│  │  ├──01_agt_00002.png
│  │  ├──[...]
│  │  └──part-00000.json
│  ├──part-00002
│  ├──part-00003
│  ├──[...]
│  └──part-00072
└──metadata.parquet
  ```
        
These sub-folders have names `part-0xxxx`, and each image has a name which has a format, `[quality]_[abbreviation]_[image number].png`. The JSON file in a sub-folder has the same name as the sub-folder. Each image is a `PNG` file. The JSON file contains key-value pairs mapping image filenames to their prompts and aesthetic scores.


## Data Instances

For example, below is the image of `01_agt_00007.png` and its key-value pair in `part-00000.json`.

<img width="300" src="https://i.imgur.com/LNNVGA2.png">

```json
{
  "01_agt_00007.png": {
        "text": "agnes tachyon \(umamusume\), labcoat, closed eyes, white background, single earring, tracen school uniform, smile, open mouth, sleeves past fingers, blush, upper body, sleeves past wrists, purple shirt, facing viewer, sailor collar, bowtie, long sleeves, :d, purple bow, white coat, breasts",
        "width": 1190,
        "height": 1684,
        "pixels": 2003960,
        "LAION_aesthetic": 6.2257309,
        "cafe_aesthetic": 0.97501057
    },
}
```  

## Data Fields

- key: Unique image name
- `text`: Manipulated tags
- `width`: Width of image
- `height`: Height of image
- `pixels`: Pixels(Width*Height) of image
- `LAION_aesthetic`: Aesthetic score by [CLIP+MLP Aesthetic Score Predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor)
- `cafe_aesthetic`: Aesthetic score by [cafe aesthetic](https://huggingface.co/cafeai/cafe_aesthetic)


## Data Metadata

To help you easily access prompts and other attributes of images without downloading all the Zip files, we include metadata table `metadata.parquet` for ULTIMA.

The shape of `metadata.parquet` is (14460, 8). We store these tables in the Parquet format because Parquet is column-based: you can efficiently query individual columns (e.g., texts) without reading the entire table.

Below are first three rows from `metadata.parquet`.

| image_name | text | part_id | width | height | pixels | LAION_aesthetic | cafe_aesthetic | 
|:--:|:---------------------------------------|:-:|:-:|:-:|:-:|:-:|:-:| 
| 01_agt_00000.png | agnes tachyon \\\(umamusume\\\), vehicle focus, motor vehicle, ground vehicle, labcoat, sleeves past wrists, sports car, sleeves past fingers, yellow sweater, black necktie, open mouth, black pantyhose, smile, looking at viewer, single earring, short necktie, holding | 0 | 3508 | 2480 | 8699840 | 5.99897194 | 0.9899081 |  
| 01_agt_00001.png | agnes tachyon \\\(umamusume\\\), labcoat, sleeves past wrists, sleeves past fingers, long sleeves, black pantyhose, skirt, smile, white background, white coat, cowboy shot, from side, profile, hand up, closed mouth, yellow sweater, collared shirt, black shirt, black necktie, pen coat, looking to the side | 0 | 1105 | 1349 | 1490645 | 6.3266325 | 0.99231464 |  
| 01_agt_00002.png | agnes tachyon \\\(umamusume\\\), labcoat, test tube, sitting, crossed legs, yellow sweater, sleeves past wrists, black pantyhose, sleeves past fingers, black necktie, boots removed, high heels, full body, long sleeves, shoes, high heel boots, single shoe, sweater vest, white coat, smile, closed mouth, collared shirt, single boot, white footwear, white background, single earring, black shirt, short necktie, open coat, vial | 0 | 2000 | 2955 | 5910000 | 6.21014023 | 0.94741267 |  
|

## Metadata Schema

|Column|Type|Description|
|:---|:---|:---|
|`image_name`|`string`| Image filename |
|`text`|`string`| The manipulated text of image for alignment |
|`part_id`|`uint16`| Folder ID of this image |
|`width`|`uint16`| Image width |
|`height`|`uint16`| Image height |
|`pixels`|`uint32`| Image pixels |
|`LAION_aesthetic`|`float32`| LATION aesthetic score of image |
|`cafe_aesthetic`|`float32`| cafe aesthetic score of image |
|


# Considerations for Using the Data  
## Limitations and Bias  
The whole process was based on the subjectivity of the author.  

 0. Domain of the dataset, which only contains characters in Uma Musume: Pretty Derby.
 1. Collection of images
 3. Calibration on images
 4. Manipulation of tags
 5. Alignment on tags
 6. Separation of images by quality  

Therefore, the dataset is totally based on author's supervision, not on any objective metric.  


## Guidelines for secondary creation of Uma Musume: Pretty Derby
Here is the guidelines for secondary creation of Uma Musume: Pretty Derby from Cygames, Inc.
>We would like to provide you with the guidelines for secondary creations of Uma Musume Pretty Derby.  
>This work features numerous characters based on real-life racehorses, and it has been made possible through the cooperation of many individuals, including the horse owners who have lent their horse names.  
>We kindly ask everyone, including fans of the racehorses that serve as motifs, horse owners, and related parties, to refrain from expressions that may cause discomfort or significantly damage the image of the racehorses or characters.  
>Specifically, please refrain from publishing creations that fall under the following provisions within Uma Musume Pretty Derby  
>1. Creations that aim to harm this work, the thoughts of third parties, or their reputation
>2. Violent, grotesque, or sexually explicit content  
>3. Creations that excessively support or denigrate specific politics, religions, or beliefs
>4. Expressions with antisocial content  
>5. Creations that infringe upon the rights of third parties
>
>These guidelines have been established after consultation with the management company responsible for the horse names.  
>In cases that fall under the aforementioned provisions, we may have to consider taking legal measures if necessary.  
>These guidelines do not deny the fan activities of those who support Uma Musume.  
>We have established these guidelines to ensure that everyone can engage in fan activities with peace of mind.  
>We appreciate your understanding and cooperation.  
>Please note that we will not provide individual responses to inquiries regarding these guidelines.  
>The Uma Musume project will continue to support racehorses and their achievements alongside everyone, in order to uphold the dignity of these renowned horses.  

Translated by ChatGPT. The original document(in japanese) is [here](https://umamusume.jp/derivativework_guidelines/).  


## Licensing Information
The dataset is made available for academic research purposes only and for non-commercial purposes. All the images are collected from the Internet, and the copyright of images belongs to the original owners. If any of the images belongs to you and you would like it removed, please inform us, we will try to remove it from the dataset.


## Citation
```bibtex
@misc{ULTIMA,
  author       = {Oh Giyeong (BootsofLagrangian), Kang Dohoon (Haken)},
  title        = {ULTIMA - Uma Musume Labeled Text-Image Alignment Dataset},
  howpublished = {\url{https://huggingface.co/datasets/UmaDiffusion/ULTIMA}},
  month        = {July},
  year         = {2023}
}
```

