task_categories:text-to-image task_ids:image-captioning task_ids:face-detection annotations_creators:no-annotation language_creators:found size_categories:n<500MB source_datasets:original language:en license:cc0-1.0 stable diffusion prompt engineering prompts research paper facial expression recognition emotion recognition region:us
---
layout: default
title: Home
nav_order: 1
has_children: false
annotations_creators:
- no-annotation
language:
- en
language_creators:
- found
pretty_name: DiffusionFER
size_categories:
- n<500MB
source_datasets:
- original
license: cc0-1.0
tags:
- stable diffusion
- prompt engineering
- prompts
- research paper
- facial expression recognition
- emotion recognition
task_categories:
- text-to-image
task_ids:
- image-captioning
- face-detection
---

## Dataset Description

- **Homepage:** [DiffusionFER homepage](https://kdhht2334.github.io/)
- **Repository:** [DiffusionFER repository](https://github.com/kdhht2334/Facial-Expression-Recognition-Zoo)
- **Distribution:** [DiffusionFER Hugging Face Dataset](https://huggingface.co/datasets/FER-Universe/DiffusionFER)
- **Point of Contact:** [Daeha Kim](mailto:kdhht5022@gmail.com)


### Summary

DiffusionFER is the large-scale text-to-image prompt database for face-related tasks. It contains about **1M(ongoing)** images generated by [Stable Diffusion](https://github.com/camenduru/stable-diffusion-webui-colab) using prompt(s) and other parameters.

DiffusionFER is available at [🤗 Hugging Face Dataset](https://huggingface.co/datasets/FER-Universe/DiffusionFER).


### Downstream Tasks and Leaderboards

This DiffusionFER dataset can be utilized for the following downstream tasks.
  - Face detection
  - Facial expression recognition
  - Text-to-emotion prompting

In addition, the virtual subjects included in this dataset provide opportunities to perform various vision tasks related to face privacy.


### Data Loading

DiffusionFER can be loaded via both Python and Git. Please refer Hugging Face [`Datasets`](https://huggingface.co/docs/datasets/quickstart).

```python
from datasets import load_dataset

dataset = load_dataset("FER-Universe/DiffusionFER")
```

```bash
git lfs install
git clone https://huggingface.co/datasets/FER-Universe/DiffusionFER
```

### Pre-trained model

You can easily download and use pre-trained __Swin Transformer__ model with the `Diffusion_Emotion_S` dataset.

Later, Transformer models with the `Diffusion_Emotion_M` or `Diffusion_Emotion_L` will be released.

```python
from transformers import AutoFeatureExtractor, AutoModelForImageClassification

extractor = AutoFeatureExtractor.from_pretrained("kdhht2334/autotrain-diffusion-emotion-facial-expression-recognition-40429105176")

model = AutoModelForImageClassification.from_pretrained("kdhht2334/autotrain-diffusion-emotion-facial-expression-recognition-40429105176")
```

Or just clone the model repo


```bash
git lfs install
git clone https://huggingface.co/kdhht2334/autotrain-diffusion-emotion-facial-expression-recognition-40429105176
```

  - Quick links: [huggingface model documentation](https://huggingface.co/docs/transformers/main/en/model_doc/swin#transformers.SwinForImageClassification)



### Sample Gallery

▼Happy

![Gallery(happy)](https://drive.google.com/uc?id=10YW9XHXFJ9cjutis9Pwpgd0ld6JI84P3)


▼Angry

![Gallery(happy)](https://drive.google.com/uc?id=14qbmOgzqqXGxkatjMfqaUmf0xYwDz--g)


### Subsets

DiffusionFER supports a total of three distinct splits. And, each split additionally provides a face region cropped by [face detector](https://github.com/timesler/facenet-pytorch).
   - DifussionEmotion_S (small), DifussionEmotion_M (medium), DifussionEmotion_L (large).

|Subset|Num of Images|Size|Image Directory |
|:--|--:|--:|--:|
|DifussionEmotion_S (original) | 1.5K | 647M | `DifussionEmotion_S/` |
|DifussionEmotion_S (cropped)  | 1.5K | 322M | `DiffusionEmotion_S_cropped/` |
|DifussionEmotion_M (original) | N/A  | N/A  | `DifussionEmotion_M/` |
|DifussionEmotion_M (cropped)  | N/A  | N/A  | `DiffusionEmotion_M_cropped/` |
|DifussionEmotion_L (original) | N/A  | N/A  | `DifussionEmotion_L/` |
|DifussionEmotion_L (cropped)  | N/A  | N/A  | `DiffusionEmotion_L_cropped/` |


## Dataset Structure

We provide DiffusionFER using a modular file structure. `DiffusionEmotion_S`, the smallest scale, contains about 1,500 images and is divided into folders of a total of 7 emotion classes. The class labels of all these images are included in `dataset_sheet.csv`.
   - In `dataset_sheet.csv`, not only 7-emotion class but also _valence-arousal_ value are annotated.

```bash
# Small version of DB
./
├── DifussionEmotion_S
│   ├── angry
│   │   ├── aaaaaaaa_6.png
│   │   ├── andtcvhp_6.png
│   │   ├── azikakjh_6.png
│   │   ├── [...]
│   ├── fear
│   ├── happy
│   ├── [...]
│   └── surprise
└── dataset_sheet.csv
```

- Middle size DB will be uploaded soon.
```bash
# Medium version of DB
(ongoing)
```

- TBD
```bash
# Large version of DB
(ongoing)
```


### Prompt Format

Basic format is as follows: "`Emotion`, `Race` `Age` style, a realistic portrait of `Style` `Gender`, upper body, `Others`".

- ex) one person, neutral emotion, white middle-aged style, a realistic portrait of man, upper body

Examples of format categories are listed in the table below.

| Category | Prompt(s) |
| --- | --- |
| `Emotion` | neutral emotion<br>happy emotion, with open mouth, smiley<br>sad emotion, with tears, lowered head, droopy eyebrows<br>surprise emotion, with open mouth, big eyes<br>fear emotion, scared, haunted<br>disgust emotion, frown, angry expression with open mouth<br>angry emotion, with open mouth, frown eyebrow, fierce, furious | 
| `Race`    | white<br>black<br>latin | 
| `Age`     | teen<br>middle-aged<br>old |
| `Gender`  | man<br>woman |
| `Style`   | gentle<br>handsome<br>pretty<br>cute<br>mature<br>punky<br>freckles<br>beautiful crystal eyes<br>big eyes<br>small nose<br>... |
| `Others`  | 4K<br>8K<br>cyberpunk<br>camping<br>ancient<br>medieval Europe<br>... |


### Prompt Engineering

You can improve the performance and quality of generating default prompts with the settings below.
   
```
{
  "negative prompt": "sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, normal quality, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, backlight, (duplicate:1.331), (morbid:1.21), (mutilated:1.21), mutated hands, (poorly drawn hands:1.331), (bad anatomy:1.21), (bad proportions:1.331), extra limbs, (disfigured:1.331), (missing arms:1.331), (extra legs:1.331), (fused fingers:1.61051), (too many fingers:1.61051), (unclear eyes:1.331), bad hands, missing fingers, extra digit",
  "steps": 50,
  "sampling method": "DPM++ 2M Karras"
  "Width": "512",
  "Height": "512",
  "CFG scale": 12.0,
  "seed": -1,
}
```

### Annotations

The DiffusionFER contains annotation process both 7-emotion classes and valence-arousal values.

#### Annotation process

This process was carried out inspired by the theory of the two research papers below.

  - JA Russell, [A circumplex model of affect](https://d1wqtxts1xzle7.cloudfront.net/38425675/Russell1980-libre.pdf?1439132613=&response-content-disposition=inline%3B+filename%3DRussell1980.pdf&Expires=1678595455&Signature=UtbPsezND6w8vbISBiuL-ECk6hDI0etLcJSE7kJMC~hAkMSu9YyQcPKdVpdHSSq7idfcQ~eEKsqptvYpy0199DX0gi-nHJwhsciahC-zgDwylEUo6ykhP6Ab8VWCOW-DM21jHNvbYLQf7Pwi66fGvm~5bAXPc1o4HHpQpk-Cr7b0tW9lYnl3qgLoVeIICg6FLu0elbtVztgH5OS1uL6V~QhiP2PCwZf~WCHuJRQrWdPt5Kuco0lsNr1Qikk1~d7HY3ZcUTRZcMNDdem8XAFDH~ak3QER6Ml~JDkNFcLuygz~tjL4CdScVhByeAuMe3juyijtBFtYWH2h30iRkUDalg__&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA)
  - A Mollahosseini et al., [AffectNet](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8013713&casa_token=C3QmhmiB6Y8AAAAA:1CiUll0bhIq06M17YwFIvxuse7GOosEN9G1A8vxVzR8Vb5eaFp6ERIjg7xhSIQlf008KLsfJ-w&tag=1)

#### Who are the annotators?

[Daeha Kim](mailto:kdhht5022@gmail.com) and [Dohee Kang](mailto:asrs777@naver.com)


## Additional Information

### Dataset Curators

DiffusionFER is created by [Daeha Kim](https://kdhht2334.github.io/) and [Dohee Kang](https://github.com/KangDohee2270).

### Acknowledgments

This repository is heavily inspired by [DiffusionDB](https://huggingface.co/datasets/poloclub/diffusiondb), with some format references. Thank you for your interest in [DiffusionDB](https://huggingface.co/datasets/poloclub/diffusiondb).

### Licensing Information

The DiffusionFER is available under the [CC0 1.0 License](https://creativecommons.org/publicdomain/zero/1.0/).

### Contributions

If you have any questions, feel free to [open an issue](https://github.com/kdhht2334/Facial-Expression-Recognition-Zoo/issues/new) or contact [Daeha Kim](https://kdhht2334.github.io/).