task_categories:text-classification task_categories:text-generation size_categories:100M<n<1B language:jp license:apache-2.0 blogs training text not-for-all-audiences region:us
---
license: apache-2.0
language:
- jp
tags:
- blogs
- training
- text
- not-for-all-audiences
task_categories:
- text-classification
- text-generation
pretty_name: BLiterature
size_categories:
- 100M<n<1B
---

# Dataset Card for BLiterature

*BLiterature is part of a bigger project that is not yet complete. Not all information here may be accurate or accessible.*

## Dataset Description

- **Homepage:** (TODO)
- **Repository:** N/A
- **Paper:** N/A 
- **Leaderboard:** N/A
- **Point of Contact:** KaraKaraWitch

### Dataset Summary

BLiterature is a raw dataset dump consisting of text from at most 260,261,224 blog posts (excluding categories and date-grouped posts) from blog.fc2.com.

### Supported Tasks and Leaderboards

This dataset is primarily intended for unsupervised training of text generation models; however, it may be useful for other purposes.

* text-classification
* text-generation

### Languages

* Japanese

## Dataset Structure

All the files are located in jsonl files that has been compressed into archives of 7z.

### Data Instances

```json
["http://1kimono.blog49.fc2.com/blog-entry-50.html", 
  "<!DOCTYPE HTML\n\tPUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\"\n\t\t\"http://www.w3.org/TR/html4/loose.dtd\">\n<!--\n<!DOCTYPE HTML\n\tPUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n\t\t\"http://www.w3.org/T... (TRUNCATED)"]
```

### Data Fields

There is only 2 fields in the list. URL and content retrieved. content retrieved may contain values which the scraper ran into issues. If so they are marked in xml such as such.  

```<?xml version="1.0" encoding="utf-8"?><error>Specifc Error</error>```

URLs may not match the final url in which the page was retrieved from. As they may be redirects present while scraping.

#### Q-Score Distribution

Not Applicable

### Data Splits

The jsonl files were split roughly every 2,500,000 posts. Allow for a slight deviation of 5000 additional posts due to how the files were saved.

## Dataset Creation

### Curation Rationale

fc2 is a Japanese blog hosting website which offers a place for anyone to host their blog on. As a result, the language used compared to other more official sources is more informal and relaxed as anyone can post whatever they personally want.


### Source Data

#### Initial Data Collection and Normalization

None. No normalization is performed as this is a raw dump of the dataset.

#### Who are the source language producers?

The authors of each blog, which may include others to post on their blog domain as well.

### Annotations

#### Annotation process

No Annotations are present.

#### Who are the annotators?

No human annotators.

### Personal and Sensitive Information

As this dataset contains information from individuals, there is a more likely chance to find personally identifiable information. However, we believe that the author has pre-vetted their posts in good faith to avoid such occurrences.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset is intended to be useful for anyone who wishes to train a model to generate "more entertaining" content.
It may also be useful for other languages depending on your language model.

### Discussion of Biases

This dataset contains real life referances and revolves around Japanese culture. As such there will be a bias towards it.

### Other Known Limitations

N/A

## Additional Information

### Dataset Curators

KaraKaraWitch

### Licensing Information

Apache 2.0, for all parts of which KaraKaraWitch may be considered authors. All other material is distributed under fair use principles.  

Ronsor Labs additionally is allowed to relicense the dataset as long as it has gone through processing.

### Citation Information

```
@misc{bliterature,
  title         = {BLiterature: fc2 blogs for the masses.},
  author        = {KaraKaraWitch},
  year          = {2023},
  howpublished  = {\url{https://huggingface.co/datasets/KaraKaraWitch/BLiterature}},
}
```

### Name Etymology

[Literature (リテラチュア) - Reina Ueda (上田麗奈)](https://www.youtube.com/watch?v=Xo1g5HWgaRA)  
`Blogs` > `B` + `Literature` > `BLiterature`

### Contributions

- [@KaraKaraWitch (Twitter)](https://twitter.com/KaraKaraWitch) for gathering this dataset.  
- [neggles (Github)](https://github.com/neggles) for providing compute for the gathering of dataset.