task_categories:text-classification task_categories:text-generation size_categories:100K<n<1M language:ja license:apache-2.0 novel training region:us
---
license: apache-2.0
language:
- ja
tags:
- novel
- training
task_categories:
- text-classification
- text-generation
pretty_name: Syosetuka ni Narou 711K
size_categories:
- 100K<n<1M
---

# Dataset Card for Syosetu711K

*The BigKnow2022 dataset and its subsets are not yet complete. Not all information here may be accurate or accessible.*

## Dataset Description

- **Homepage:** (TODO)
- **Repository:** <https://github.com/RyokoAI/BigKnow2022>
- **Paper:** N/A 
- **Leaderboard:** N/A
- **Point of Contact:** Ronsor/undeleted <ronsor@ronsor.com>

### Dataset Summary

Syosetu711K is a dataset composed of approximately 711,700 novels scraped from the Japanese novel self-publishing
website Syosetuka ni Narou (JA: 小説家になろう, lit. "Let's Become a Novelist") between March 26 and March 27, 2023.
The dataset contains most if not all novels published on the site, regardless of length or quality; however, we
include metadata so users of this dataset can filter and evaluate its contents.

Syosetu711Kは、日本の小説投稿サイト「小説家になろう」から2023年3月26日から27日にかけてスクレイプされた約711,700冊の小説から
構成されるデータセットです。このデータセットには、長さや品質に関係なく、サイトに掲載されているほとんどの小説が含まれています。ただし、
各小説のIDも含まれているため、小説家になろうAPIを使ってその情報を検索することができます。

### Supported Tasks and Leaderboards

This dataset is primarily intended for unsupervised training of text generation models; however, it may be useful for other purposes.

* text-classification
* text-generation

### Languages

* Japanese

## Dataset Structure

### Data Instances

```json
{
  "text": "【小説タイトル】\n焼けて爛れる恋よりも、微睡む優しい愛が欲しい\n【Ｎコード】\nＮ５０２９ＩＤ\n【作者名】\n秋暁秋季\n【あらすじ】\n俺の彼女は物凄く気の多い人だった。\nお眼鏡に適う奴が居れば、瞳孔を蕩
けさせる人だった。\nその癖照れ屋で、すぐに目を逸らす。\nな...",
  "meta": {
    "subset": "syosetu",
    "q": 0.6,
    "id": "N5029ID",
    "author": "秋暁秋季",
    "userid": 719797,
    "title": "焼けて爛れる恋よりも、微睡む優しい愛が欲しい",
    "length": 871,
    "points": 0,
    "lang": "ja",
    "chapters": 1,
    "keywords": ["気が多い", "浮気性", "無愛想", "照れる", "嫉妬", "好みではない", "クソデカ感情", "空気のような安心感"],
    "isr15": 0,
    "genre": 102,
    "biggenre": 1
  }
}
{
  "text": "【小説タイトル】\n【能力者】\n【Ｎコード】\nＮ９８６４ＩＢ\n【作者名】\n夢音いちご\n【あらすじ】\n私立アビリティ学園。\n小・中・高・大が一貫となった、大規模な名門校。\nそして、ここは規模の大きさだけ
でなく、ある特殊な制度を設けて\nいることでも有名だ。\nそれ...",
  "meta": {
    "subset": "syosetu",
    "q": 0.6,
    "id": "N9864IB",
    "author": "夢音いちご",
    "userid": 1912777,
    "title": "【能力者】",
    "length": 2334,
    "points": 0,
    "lang": "ja",
    "chapters": 2,
    "keywords": ["ガールズラブ", "身分差", "伝奇", "日常", "青春", "ラブコメ", "女主人公", "学園", "魔法", "超能力"],
    "isr15": 0,
    "genre": 202,
    "biggenre": 2
  }
}
```

### Data Fields

* `text`: the actual novel text, all chapters
* `meta`: novel metadata
  * `subset`: dataset tag: `syosetu`
  * `lang`: dataset language: `ja` (Japanese)
  * `id`: novel ID/ncode
  * `author`: author name
  * `userid`: author user ID
  * `title`: novel title
  * `length`: novel length in words
  * `points`: global points (corresponds to `global_point` from the Syosetu API)
  * `q`: q-score (quality score) calculated based on `points`
  * `chapters`: number of chapters (corresponds to `general_all_no` from the Syosetu API)
  * `keywords`: array of novel keywords (corresponds to `keyword` from the Syosetu API, split on spaces)
  * `isr15`: whether the novel is rated R15+
  * `genre`: novel genre ID (optional, see Syosetu API documentation)
  * `biggenre`: general novel genre ID (optional, see Syosetu API documentation)
  * `isr18`: whether the novel is rated R18+
  * `nocgenre`: novel genre ID (optional, only available if `isr18` is true, see Syosetu API documentation)
 
*For further reference, see the Syosetuka ni Narou API documentation: <https://dev.syosetu.com/man/api/> (JA).*

#### Q-Score Distribution

```
0.00: 0
0.10: 0
0.20: 0
0.30: 0
0.40: 0
0.50: 213005
0.60: 331393
0.70: 101971
0.80: 63877
0.90: 1542
1.00: 2
```

### Data Splits

No splitting of the data was performed.

## Dataset Creation

### Curation Rationale

Syosetuka ni Narou is the most popular website in Japan for authors wishing to self-publish their novels online. Many works on
the site been picked up by large commercial publishers. Because of this, we believe that this dataset provides a large corpus
of high-quality, creative content in the Japanese language.

### Source Data

#### Initial Data Collection and Normalization

*More information about any referenced scripts, commands, or programs used may be found in the BigKnow2022 GitHub repository.*

First, metadata for all novels on the site was gathered into a JSON lines (JSONL) file. The Syosetuka ni Narou API was used to
obtain this information.

Second, this listing was used to create a secondary text file containing a list of only the novel "ncodes," or IDs. This
secondary file was distributed to downloader nodes.

Third, the sister site <https://pdfnovels.net> was queried with each novel ID, and the resulting PDF was saved for later processing.

Fourth, the `pdftotext` tool was used to convert the PDF files to text documents. A few other scripts were then used to clean up
the resulting text files.

Finally, the text files and other metadata were converted into the specified data field schema above, and the resulting JSON entries
were concatenated into the Syosetu711K dataset. The version uploaded to this repository, however, is split into multiple files,
numbered 00 through 20 inclusive.

#### Who are the source language producers?

The authors of each novel.

### Annotations

#### Annotation process

Titles and general genre were collected alongside the novel text and IDs.

#### Who are the annotators?

There were no human annotators.

### Personal and Sensitive Information

The dataset contains only works of fiction, and we do not believe it contains any PII.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset is intended to be useful for anyone who wishes to train a model to generate "more entertaining" content in Japanese.
It may also be useful for other languages depending on your language model.

### Discussion of Biases

This dataset is composed of fictional works by various authors. Because of this fact, the contents of this dataset will reflect
the biases of those authors. **Additionally, this dataset contains NSFW material and was not filtered. Beware of stereotypes.**

### Other Known Limitations

N/A

## Additional Information

### Dataset Curators

Ronsor Labs

### Licensing Information

Apache 2.0, for all parts of which Ronsor Labs or the Ryoko AI Production Committee may be considered authors. All other material is
distributed under fair use principles.

### Citation Information

```
@misc{ryokoai2023-bigknow2022,
  title         = {BigKnow2022: Bringing Language Models Up to Speed},
  author        = {Ronsor},
  year          = {2023},
  howpublished  = {\url{https://github.com/RyokoAI/BigKnow2022}},
}
```

### Contributions

Thanks to @ronsor (GH) for gathering this dataset.