task_categories:text-generation task_categories:fill-mask language:en license:cc-by-sa-3.0 language-modeling masked-language-modeling region:us
---
language_bcp47:
- nds-nl
license: cc-by-sa-3.0
language:
- en
task_categories:
- text-generation
- fill-mask
tags:
- language-modeling
- masked-language-modeling
pretty_name: SuperWIKI
---


# Dataset Card for SuperWIKI

## Dataset Description

- **Homepage:** (TODO)
- **Repository:** N/A
- **Paper:** N/A
- **Leaderboard:** N/A
- **Point of Contact:** KaraKaraWitch

### Dataset Summary

> If you show most of those to people and ask them to form an opinion,  
> the answer isn't just going to be "I don't know": it'll be "I don't care." 
>  - [Tom Scott](https://www.youtube.com/watch?v=ALy6e7GbDRQ&t=90s)
>

SuperWIKI is a focused dataset on wikipedia articles.

### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

- English

## Dataset Structure

All the files are located in gzip'd jsonl files..

### Data Instances

Refer to [Wikimedia Enterprise (API Data Dictionary)](https://enterprise.wikimedia.com/docs/data-dictionary/) for more data instances.

### Data Fields

<TBD>

#### Q-Score Distribution

Not Applicable

### Data Splits

No data splits were done.

## Dataset Creation

### Curation Rationale

"Wikipedia is a wonderful resources however it could be considered too sparse as there are many articles that are not important for the common user..."

> The abundance of less significant or obscure topics can also contribute to the perceived sparsity. While Wikipedia's commitment to covering even niche subjects is commendable, it might be overwhelming for casual users seeking concise and essential information. For instance, niche historical events, minor fictional characters, or obscure scientific theories might exist as standalone articles, but their relevance to the everyday reader could be questioned. - *ChatGPT*

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

KaraKaraWitch

### Licensing Information

Most of Wikipedia's text and many of its images are co-licensed under the
[Creative Commons Attribution-ShareAlike 3.0 Unported License](https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License)
(CC BY-SA) and the [GNU Free Documentation License](https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_Documentation_License)
(GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts). 

Some text has been imported only under CC BY-SA and CC BY-SA-compatible license and cannot be reused under GFDL; such
text will be identified on the page footer, in the page history, or on the discussion page of the article that utilizes
the text.

### Citation Information

```
@misc{superwiki,
  title         = {SuperWIKI: Wikipedia for commoners.},
  author        = {KaraKaraWitch},
  year          = {2023},
  howpublished  = {\url{https://huggingface.co/datasets/RyokoExtra/SuperWIKI}},
}
```

### Name Etymology

N/A

### Contributions

- [@KaraKaraWitch (Twitter)](https://twitter.com/KaraKaraWitch) for gathering this dataset.
- [@sirneggles (Twitter)](https://twitter.com/sirneggles) for provided compute.