task_categories:text-classification task_categories:text-generation size_categories:10M<n<100M language:en license:cc-by-sa-3.0 wiki training region:us
---
license: cc-by-sa-3.0
language:
- en
tags:
- wiki
- training
task_categories:
- text-classification
- text-generation
pretty_name: Fandom23K Wikis
size_categories:
- 10M<n<100M
---

# Dataset Card for Fandom23K

*The BigKnow2022 dataset and its subsets are not yet complete. Not all information here may be accurate or accessible.*

## Dataset Description

- **Homepage:** (TODO) https://docs.ryokoai.com/docs/training/dataset#Fandom22K 
- **Repository:** <https://github.com/RyokoAI/BigKnow2022>
- **Paper:** N/A 
- **Leaderboard:** N/A
- **Point of Contact:** Ronsor/undeleted <ronsor@ronsor.com>

### Dataset Summary

Fandom23K is a dataset composed of 15,616,749 articles scraped from approximately 23,665 Fandom.com wikis between March 14 and March 18, 2023.
It is a subset of the upcoming BigKnow2022 dataset.

### Supported Tasks and Leaderboards

This dataset is primarily intended for unsupervised training of text generation models; however, it may be useful for other purposes.

* text-classification

### Languages

* English
* Potentially other languages in much smaller quantities.

## Dataset Structure

### Data Instances

```json
{
  "tag": "fandom.wikia2011",
  "text": "# Add Your Wiki's Highlights\n\nWrite the text of your article here!-_-\n\n",
  "title": "Add Your Wiki's Highlights"
}
{
  "tag": "fandom.wikia2011",
  "text": "# Add Your Wiki's Highlights!\n\nWikia wants to hear from you! What significant milestones did your wiki experience in 2011? What cool things did the community try out?\nCreate a page for the wiki you're most active on! Be sure to add it to the Entertainment, Gaming, or Lifestyle categories so it shows up in the right place!\n\n",
  "title": "Add Your Wiki's Highlights!"
}
{
  "tag": "fandom.wikia2011",
  "text": "# Assassins Creed Wiki 2011\n\nIn 2011, Assassin's Creed Wiki tested new Wikia features such as Message Wall, Chat, and New Layouts.\n\n",
  "title": "Assassins Creed Wiki 2011"
}
```

### Data Fields

* **text**: the actual article text
* **title**: the article title
* **tag**: text source tag, in the following format: `fandom.<wiki name>`

### Data Splits

No splitting of the data was performed.

## Dataset Creation

### Curation Rationale

Fandom23K provides an up-to-date corpus containing pop culture and media information spanning a variety of interests and
hobbies. Previous datasets containing such information are either part of a large and harder-to-handle whole, such as
Common Crawl, do not provide enough variety, or are simply outdated.

### Source Data

#### Initial Data Collection and Normalization

*More information about any referenced scripts, commands, or programs used may be found in the BigKnow2022 GitHub repository.*

First, a list of active Fandom wikis was gathered into a text file. Active is defined as "having at least 250 images on the wiki."
This list was gathered in early January 2023, despite the actual wiki content being more recent.

Second, the `scrape_fandom.py` script was used to generate and download an up to date dump for each of the wikis.

Third, `wikiextractor` was used to process these dumps into single XML files containing each article stripped of all formatting
besides links.

Fourth, `dump2jsonl` was used to convert the XML files into JSONL files with an article per line. Light markdown formatting was
applied, converting the HTML links to markdown-formatted links, and automatically making the article's title a header.

Finally, the JSONL files were concatenated into the Fandom23K dataset. The version uploaded to this repository, however, is split
into multiple files, numbered 00 through 04 inclusive.

#### Who are the source language producers?

The contributors of each wiki.

### Annotations

#### Annotation process

Wiki names and article titles were collected alongside the article text. Other than that automated process, no annotation was performed.

#### Who are the annotators?

There were no human annotators.

### Personal and Sensitive Information

The dataset was collected from public wiki data. As a result, we do not believe
it should contain any PII and did not inspect it further.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset is intended to be useful for anyone who wishes to train a model to generate "more entertaining" content requiring
knowledge of popular culture or a particular niche.

### Discussion of Biases

This dataset contains text from random Internet users and generally should not be used as an authoritative source of information.
Additionally, this dataset was not filtered at all. We recommmend its usage for research purposes only.

### Other Known Limitations

This dataset is based on a list of active wikis from January 2023, even though the actual wiki content may be more recent. Additionally,
smaller yet still active wikis may have been excluded.

## Additional Information

### Dataset Curators

Ronsor Labs

### Licensing Information

CC-BY-SA 3.0, except for any portions which state otherwise.

### Citation Information

```
@misc{ryokoai2023-bigknow2022,
  title         = {BigKnow2022: Bringing Language Models Up to Speed},
  author        = {Ronsor},
  year          = {2023},
  howpublished  = {\url{https://github.com/RyokoAI/BigKnow2022}},
}
```

### Contributions

Thanks to @ronsor for gathering this dataset.
