task_categories:image-classification task_categories:tabular-classification size_categories:100K<n<1M art region:us
---
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: test
    path: data/test-*
  - split: validation
    path: data/validation-*
dataset_info:
  features:
  - name: tag_probs
    sequence: float32
  - name: class
    dtype:
      class_label:
        names:
          '0': not_bookmarked
          '1': bookmarked_public
          '2': bookmarked_private
  splits:
  - name: train
    num_bytes: 4301053452
    num_examples: 179121
  - name: test
    num_bytes: 1433684484
    num_examples: 59707
  - name: validation
    num_bytes: 1433708496
    num_examples: 59708
  download_size: 7351682183
  dataset_size: 7168446432
task_categories:
- image-classification
- tabular-classification
tags:
- art
size_categories:
- 100K<n<1M
---

The dataset for training classification model of pixiv artworks by my preference.

## Schema

* tag_probs: List of probabilities for each tag. Preprocessed by [RF5/danbooru-pretrained](https://github.com/RF5/danbooru-pretrained) model. The index of each probability corresponds to the index of the tag in the [class_names_6000.json](https://github.com/RF5/danbooru-pretrained/blob/master/config/class_names_6000.json) file.
* class:
  * not_bookmarked (0): Generated from images randomly-sampled from [animelover/danbooru2022](https://huggingface.co/datasets/animelover/danbooru2022) dataset. The images are filtered in advance to the post with pixiv source.
  * bookmarked_public (1): Generated from publicly bookmarked images of [hakatashi](https://twitter.com/hakatashi).
  * bookmarked_private (2): Generated from privately bookmarked images of [hakatashi](https://twitter.com/hakatashi).

## Stats

train:test:validation = 6:2:2

* not_bookmarked (0): 202,290 images
* bookmarked_public (1): 73,587 images
* bookmarked_private (2): 22,659 images

## Usage

```
>>> from datasets import load_dataset

>>> dataset = load_dataset("hakatashi/hakatashi-pixiv-bookmark-deepdanbooru")
>>> dataset
DatasetDict({
    test: Dataset({
        features: ['tag_probs', 'class'],
        num_rows: 59707
    })
    train: Dataset({
        features: ['tag_probs', 'class'],
        num_rows: 179121
    })
    validation: Dataset({
        features: ['tag_probs', 'class'],
        num_rows: 59708
    })
})
>>> dataset['train'].features
{'tag_probs': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None),
 'class': ClassLabel(names=['not_bookmarked', 'bookmarked_public', 'bookmarked_private'], id=None)}
```