task_categories:text-generation task_categories:fill-mask language:en license:cc-by-sa-3.0 language-modeling masked-language-modeling region:us
---
license: cc-by-sa-3.0
language:
- en
task_categories:
- text-generation
- fill-mask
tags:
- language-modeling
- masked-language-modeling
pretty_name: SuperWIKI Cleaned
configs:
- config_name: default
  default: true
  data_files:
  - split: lang50NightShade
    path:
    - "*-lang50NightShade-*.json.gz"
  - split: lang50
    path:
    - "*-lang50-*.json.gz"
  - split: lang25
    path:
    - "*-lang25-*.json.gz"
---


# Dataset Card for SuperWIKI Cleaned

## Dataset Description

- **Homepage:** (TODO)
- **Repository:** N/A
- **Paper:** N/A
- **Leaderboard:** N/A
- **Point of Contact:** KaraKaraWitch

### Dataset Summary

> If you show most of those to people and ask them to form an opinion,  
> the answer isn't just going to be "I don't know": it'll be "I don't care." 
>  - [Tom Scott](https://www.youtube.com/watch?v=ALy6e7GbDRQ&t=90s)
>

SuperWIKI Cleaned is a focused dataset on wikipedia articles. 
This dataset is derived from raw files provided in [SuperWIKI](https://huggingface.co/datasets/RyokoExtra/SuperWIKI).

### Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

### Languages

- English

## Dataset Structure

All the files are located in gzip'd jsonl files.

### Data Instances

Refer to this sample to see all the fields:

```json
{
    "id": 35507,
    "text": "In computer network communications, the **HTTP 404**, **404 not found**, **404**, **404 error**, **page not found** or **file not found** error message is a hypertext transfer protocol (HTTP) standard response code, to indicate that the browser was able to communicate with a given server, but the server could not find what was requested. The error may also be used when a server does not wish to disclose whether it has the requested information.<TRUNCATED>",
    "title": "HTTP 404",
    "url": "https://en.wikipedia.org/wiki/HTTP_404",
    "filters": {
        "issues": [],
        "selectors": [],
        "templates": [
            "template:http",
            "template:redirect",
            "template:use dmy dates",
            "template:cite book",
            "template:portal",
            "template:anchor",
            "template:pp-move-indef",
            "template:cite news",
            "template:reflist",
            "template:short description",
            "template:citation",
            "template:error messages",
            "template:pp-semi-indef",
            "template:cite journal",
            "template:cite web"
        ],
        "rituals": []
    },
    "infobox_html": [],
    "figures_dict": [
        {
            "file_url": "./File:Wikipedia_404_Page.png",
            "caption": "English Wikipedia's 404 Page"
        },
        {
            "file_url": "./File:Wikimedia_error_404.png",
            "caption": "The Wikimedia 404 message"
        }
    ]
}
```

### Data Fields

`id`: The article ID in question
`text`: The HTML Text (After post-processing) from SuperWIKI converted to markdown with links removed and formatting (Bold, italics) kept.  
`title`: The title of the wikipedia article.  
`url`: The URL of the article.  
`filters`: Metadata of filters found/used in the dataset.  
 - `issues`: A list of custom list of templates that has been removed from the html (ala, pre-processing) for the article.  
 - `selectors`: `issues` are based on templates, which may have multiple templates but mean the same thing. In that case, the selectors provide a unduplicated css class selectors that were used for the article. (`Template:Few sources` is the same as `Template:More citations needed` for example.)  
 - `rituals`: List of "Rituals" used to remove even more "Issue" templates. If not present, this field is empty.
 - `templates`: Used for debugging but are all the templates found in the article.  

`infobox_html`: A list of side infoboxes that ae extracted out from the text.  
`figures_dict`: A list of figures used in the article. Again, extracted out from the text.  

#### Q-Score Distribution

Not Applicable

### Data Splits

No data splits were done.

## Dataset Creation

### Curation Rationale

"Wikipedia is a wonderful resources however it could be considered too sparse as there are many articles that are not important for the common user..."

> The abundance of less significant or obscure topics can also contribute to the perceived sparsity. While Wikipedia's commitment to covering even niche subjects is commendable, it might be overwhelming for casual users seeking concise and essential information. For instance, niche historical events, minor fictional characters, or obscure scientific theories might exist as standalone articles, but their relevance to the everyday reader could be questioned. - *ChatGPT*

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

This article removes all "Notices" from all articles to provide a cleaner version of wikipedia.  
You should consider adding flags back into the dataset if you want to tell the user about the potential issues.

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

KaraKaraWitch

### Licensing Information

Most of Wikipedia's text and many of its images are co-licensed under the
[Creative Commons Attribution-ShareAlike 3.0 Unported License](https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License)
(CC BY-SA) and the [GNU Free Documentation License](https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_Documentation_License)
(GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts). 

Some text has been imported only under CC BY-SA and CC BY-SA-compatible license and cannot be reused under GFDL; such
text will be identified on the page footer, in the page history, or on the discussion page of the article that utilizes
the text.

### Citation Information

```
@misc{superwiki,
  title         = {SuperWIKI Cleaned: Wikipedia for commoners.},
  author        = {KaraKaraWitch},
  year          = {2023},
  howpublished  = {\url{https://huggingface.co/datasets/RyokoExtra/SuperWIKI}},
}
```

### Name Etymology

N/A

### Contributions

- [@KaraKaraWitch (Twitter)](https://twitter.com/KaraKaraWitch) for gathering this dataset.
- [@sirneggles (Twitter)](https://twitter.com/sirneggles) for provided compute.