task_categories:text-generation task_categories:text-classification size_categories:10K<n<100K language:ja license:cc-by-4.0 region:us
---
license: cc-by-4.0
task_categories:
- text-generation
- text-classification
language:
- ja
size_categories:
- 10K<n<100K
---

# Overview
This dataset provides a convenient and user-friendly format of data from [Aozora Bunko (青空文庫)](https://www.aozora.gr.jp/), a website that compiles public-domain books in Japan, ideal for Machine Learning applications.

# Methodology

The code to reproduce this dataset is made available on GitHub: [globis-org/aozorabunko-exctractor](https://github.com/globis-org/aozorabunko-extractor).

## 1. Data collection
We firstly downloaded the [CSV file that lists all works](https://www.aozora.gr.jp/index_pages/person_all.html). The information extracted from this CSV is incorporated into the `meta` field.
Next, we filtered out any books not categorized as public domain.
We retrieved the main text of each book corresponding to every row in the CSV and incorporated it into the `text` field.

## 2. Deduplication
We removed entries where the `図書カードURL` (Library card URL) in this CSV did not match with the `作品ID` (Work ID) and `人物ID` (Person ID).
In addition, entries with text identical to previously encountered text were discarded.

## 3. Cleaning
The data in the `text` field was then cleaned in the following sequence:

1. Convert new lines to `\n`
2. Remove headers
3. Remove footnotes and add them to the `footnote` field
4. Remove ruby (phonetic guides)
5. Convert specific characters, such as external characters and iteration marks, into standard Unicode characters
6. Convert inserted notes into regular parenthetical text
7. Remove any remaining markup
8. Remove leading and trailing new lines and horizontal rules

# Tips

If you prefer to employ only modern Japanese, you can filter entries with: `row["meta"]["文字遣い種別"] == "新字新仮名"`.

# License
CC BY 4.0