task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:found multilinguality:multilingual size_categories:n<1K size_categories:1K<n<10K size_categories:10K<n<100K size_categories:100K<n<1M size_categories:1M<n<10M size_categories:10M<n<100M size_categories:100M<n<1B size_categories:1B<n<10B source_datasets:original language:af language:am language:ar language:az language:be language:bg language:bn language:ca language:ceb language:co language:cs language:cy language:da language:de language:el language:en language:eo language:es language:et language:eu language:fa language:fi language:fil language:fr language:fy language:ga language:gd language:gl language:gu language:ha language:haw language:he language:hi language:hmn language:ht language:hu language:hy language:id language:ig language:is language:it language:iw language:ja language:jv language:ka language:kk language:km language:kn language:ko language:ku language:ky language:la language:lb language:lo language:lt language:lv language:mg language:mi language:mk language:ml language:mn language:mr language:ms language:mt language:my language:ne language:nl language:no language:ny language:pa language:pl language:ps language:pt language:ro language:ru language:sd language:si language:sk language:sl language:sm language:sn language:so language:sq language:sr language:st language:su language:sv language:sw language:ta language:te language:tg language:th language:tr language:uk language:und language:ur language:uz language:vi language:xh language:yi language:yo language:zh language:zu license:odc-by arxiv:1910.10683 region:us
A colossal, cleaned version of Common Crawl's web crawl corpus.

Based on Common Crawl dataset: "https://commoncrawl.org".

This is the processed version of Google's mC4 dataset by AllenAI.
@article{2019t5,
    author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
    title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
    journal = {arXiv e-prints},
    year = {2019},
    archivePrefix = {arXiv},
    eprint = {1910.10683},
}
---
pretty_name: mC4
annotations_creators:
- no-annotation
language_creators:
- found
language:
- af
- am
- ar
- az
- be
- bg
- bn
- ca
- ceb
- co
- cs
- cy
- da
- de
- el
- en
- eo
- es
- et
- eu
- fa
- fi
- fil
- fr
- fy
- ga
- gd
- gl
- gu
- ha
- haw
- he
- hi
- hmn
- ht
- hu
- hy
- id
- ig
- is
- it
- iw
- ja
- jv
- ka
- kk
- km
- kn
- ko
- ku
- ky
- la
- lb
- lo
- lt
- lv
- mg
- mi
- mk
- ml
- mn
- mr
- ms
- mt
- my
- ne
- nl
- 'no'
- ny
- pa
- pl
- ps
- pt
- ro
- ru
- sd
- si
- sk
- sl
- sm
- sn
- so
- sq
- sr
- st
- su
- sv
- sw
- ta
- te
- tg
- th
- tr
- uk
- und
- ur
- uz
- vi
- xh
- yi
- yo
- zh
- zu
language_bcp47:
- bg-Latn
- el-Latn
- hi-Latn
- ja-Latn
- ru-Latn
- zh-Latn
license:
- odc-by
multilinguality:
- multilingual
size_categories:
- n<1K
- 1K<n<10K
- 10K<n<100K
- 100K<n<1M
- 1M<n<10M
- 10M<n<100M
- 100M<n<1B
- 1B<n<10B
source_datasets:
- original
task_categories:
- text-generation
- fill-mask
task_ids:
- language-modeling
- masked-language-modeling
paperswithcode_id: mc4
---

# Dataset Card for mC4

## Table of Contents

- [Dataset Card for mC4](#dataset-card-for-mc4)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
  - [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)
      - [Who are the source language producers?](#who-are-the-source-language-producers)
    - [Annotations](#annotations)
      - [Annotation process](#annotation-process)
      - [Who are the annotators?](#who-are-the-annotators)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
  - [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://huggingface.co/datasets/allenai/c4
- **Paper:** https://arxiv.org/abs/1910.10683

### Dataset Summary

A multilingual colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org".

This is the version prepared by AllenAI, hosted at this address: https://huggingface.co/datasets/allenai/c4

108 languages are available and are reported in the table below.

Note that the languages that end with "-Latn" are simply romanized variants, i.e. written using the Latin script.

| language code   | language name        |
|:----------------|:---------------------|
| af              | Afrikaans            |
| am              | Amharic              |
| ar              | Arabic               |
| az              | Azerbaijani          |
| be              | Belarusian           |
| bg              | Bulgarian            |
| bg-Latn         | Bulgarian (Latin)    |
| bn              | Bangla               |
| ca              | Catalan              |
| ceb             | Cebuano              |
| co              | Corsican             |
| cs              | Czech                |
| cy              | Welsh                |
| da              | Danish               |
| de              | German               |
| el              | Greek                |
| el-Latn         | Greek (Latin)        |
| en              | English              |
| eo              | Esperanto            |
| es              | Spanish              |
| et              | Estonian             |
| eu              | Basque               |
| fa              | Persian              |
| fi              | Finnish              |
| fil             | Filipino             |
| fr              | French               |
| fy              | Western Frisian      |
| ga              | Irish                |
| gd              | Scottish Gaelic      |
| gl              | Galician             |
| gu              | Gujarati             |
| ha              | Hausa                |
| haw             | Hawaiian             |
| hi              | Hindi                |
| hi-Latn         | Hindi (Latin script) |
| hmn             | Hmong, Mong          |
| ht              | Haitian              |
| hu              | Hungarian            |
| hy              | Armenian             |
| id              | Indonesian           |
| ig              | Igbo                 |
| is              | Icelandic            |
| it              | Italian              |
| iw              | former Hebrew        |
| ja              | Japanese             |
| ja-Latn         | Japanese (Latin)     |
| jv              | Javanese             |
| ka              | Georgian             |
| kk              | Kazakh               |
| km              | Khmer                |
| kn              | Kannada              |
| ko              | Korean               |
| ku              | Kurdish              |
| ky              | Kyrgyz               |
| la              | Latin                |
| lb              | Luxembourgish        |
| lo              | Lao                  |
| lt              | Lithuanian           |
| lv              | Latvian              |
| mg              | Malagasy             |
| mi              | Maori                |
| mk              | Macedonian           |
| ml              | Malayalam            |
| mn              | Mongolian            |
| mr              | Marathi              |
| ms              | Malay                |
| mt              | Maltese              |
| my              | Burmese              |
| ne              | Nepali               |
| nl              | Dutch                |
| no              | Norwegian            |
| ny              | Nyanja               |
| pa              | Punjabi              |
| pl              | Polish               |
| ps              | Pashto               |
| pt              | Portuguese           |
| ro              | Romanian             |
| ru              | Russian              |
| ru-Latn         | Russian (Latin)      |
| sd              | Sindhi               |
| si              | Sinhala              |
| sk              | Slovak               |
| sl              | Slovenian            |
| sm              | Samoan           |
| sn              | Shona                |
| so              | Somali               |
| sq              | Albanian             |
| sr              | Serbian              |
| st              | Southern Sotho       |
| su              | Sundanese            |
| sv              | Swedish              |
| sw              | Swahili              |
| ta              | Tamil                |
| te              | Telugu               |
| tg              | Tajik                |
| th              | Thai                 |
| tr              | Turkish              |
| uk              | Ukrainian            |
| und             | Unknown language     |
| ur              | Urdu                 |
| uz              | Uzbek                |
| vi              | Vietnamese           |
| xh              | Xhosa                |
| yi              | Yiddish              |
| yo              | Yoruba               |
| zh              | Chinese              |
| zh-Latn         | Chinese (Latin)      |
| zu              | Zulu                 |

You can load the mC4 subset of any language like this:

```python
from datasets import load_dataset

en_mc4 = load_dataset("mc4", "en")
```

And if you can even specify a list of languages:

```python
from datasets import load_dataset

mc4_subset_with_five_languages = load_dataset("mc4", languages=["en", "fr", "es", "de", "zh"])
```

### Supported Tasks and Leaderboards

mC4 is mainly intended to pretrain language models and word representations.

### Languages

The dataset supports 108 languages.

## Dataset Structure

### Data Instances

An example form the `en` config is:

```
{'timestamp': '2018-06-24T01:32:39Z',
 'text': 'Farm Resources in Plumas County\nShow Beginning Farmer Organizations & Professionals (304)\nThere are 304 resources serving Plumas County in the following categories:\nMap of Beginning Farmer Organizations & Professionals serving Plumas County\nVictoria Fisher - Office Manager - Loyalton, CA\nAmy Lynn Rasband - UCCE Plumas-Sierra Administrative Assistant II - Quincy , CA\nShow Farm Income Opportunities Organizations & Professionals (353)\nThere are 353 resources serving Plumas County in the following categories:\nFarm Ranch And Forest Retailers (18)\nMap of Farm Income Opportunities Organizations & Professionals serving Plumas County\nWarner Valley Wildlife Area - Plumas County\nShow Farm Resources Organizations & Professionals (297)\nThere are 297 resources serving Plumas County in the following categories:\nMap of Farm Resources Organizations & Professionals serving Plumas County\nThere are 57 resources serving Plumas County in the following categories:\nMap of Organic Certification Organizations & Professionals serving Plumas County',
 'url': 'http://www.californialandcan.org/Plumas/Farm-Resources/'}
```

### Data Fields

The data have several fields:

- `url`: url of the source as a string
- `text`: text content as a string
- `timestamp`: timestamp as a string

### Data Splits

To build mC4, the authors used [CLD3](https://github.com/google/cld3) to identify over 100 languages. The resulting mC4 subsets for each language are reported in this table:

| config   | train   | validation   |
|:---------|:--------|:-------------|
| af       | ?       | ?            |
| am       | ?       | ?            |
| ar       | ?       | ?            |
| az       | ?       | ?            |
| be       | ?       | ?            |
| bg       | ?       | ?            |
| bg-Latn  | ?       | ?            |
| bn       | ?       | ?            |
| ca       | ?       | ?            |
| ceb      | ?       | ?            |
| co       | ?       | ?            |
| cs       | ?       | ?            |
| cy       | ?       | ?            |
| da       | ?       | ?            |
| de       | ?       | ?            |
| el       | ?       | ?            |
| el-Latn  | ?       | ?            |
| en       | ?       | ?            |
| eo       | ?       | ?            |
| es       | ?       | ?            |
| et       | ?       | ?            |
| eu       | ?       | ?            |
| fa       | ?       | ?            |
| fi       | ?       | ?            |
| fil      | ?       | ?            |
| fr       | ?       | ?            |
| fy       | ?       | ?            |
| ga       | ?       | ?            |
| gd       | ?       | ?            |
| gl       | ?       | ?            |
| gu       | ?       | ?            |
| ha       | ?       | ?            |
| haw      | ?       | ?            |
| hi       | ?       | ?            |
| hi-Latn  | ?       | ?            |
| hmn      | ?       | ?            |
| ht       | ?       | ?            |
| hu       | ?       | ?            |
| hy       | ?       | ?            |
| id       | ?       | ?            |
| ig       | ?       | ?            |
| is       | ?       | ?            |
| it       | ?       | ?            |
| iw       | ?       | ?            |
| ja       | ?       | ?            |
| ja-Latn  | ?       | ?            |
| jv       | ?       | ?            |
| ka       | ?       | ?            |
| kk       | ?       | ?            |
| km       | ?       | ?            |
| kn       | ?       | ?            |
| ko       | ?       | ?            |
| ku       | ?       | ?            |
| ky       | ?       | ?            |
| la       | ?       | ?            |
| lb       | ?       | ?            |
| lo       | ?       | ?            |
| lt       | ?       | ?            |
| lv       | ?       | ?            |
| mg       | ?       | ?            |
| mi       | ?       | ?            |
| mk       | ?       | ?            |
| ml       | ?       | ?            |
| mn       | ?       | ?            |
| mr       | ?       | ?            |
| ms       | ?       | ?            |
| mt       | ?       | ?            |
| my       | ?       | ?            |
| ne       | ?       | ?            |
| nl       | ?       | ?            |
| no       | ?       | ?            |
| ny       | ?       | ?            |
| pa       | ?       | ?            |
| pl       | ?       | ?            |
| ps       | ?       | ?            |
| pt       | ?       | ?            |
| ro       | ?       | ?            |
| ru       | ?       | ?            |
| ru-Latn  | ?       | ?            |
| sd       | ?       | ?            |
| si       | ?       | ?            |
| sk       | ?       | ?            |
| sl       | ?       | ?            |
| sm       | ?       | ?            |
| sn       | ?       | ?            |
| so       | ?       | ?            |
| sq       | ?       | ?            |
| sr       | ?       | ?            |
| st       | ?       | ?            |
| su       | ?       | ?            |
| sv       | ?       | ?            |
| sw       | ?       | ?            |
| ta       | ?       | ?            |
| te       | ?       | ?            |
| tg       | ?       | ?            |
| th       | ?       | ?            |
| tr       | ?       | ?            |
| uk       | ?       | ?            |
| und      | ?       | ?            |
| ur       | ?       | ?            |
| uz       | ?       | ?            |
| vi       | ?       | ?            |
| xh       | ?       | ?            |
| yi       | ?       | ?            |
| yo       | ?       | ?            |
| zh       | ?       | ?            |
| zh-Latn  | ?       | ?            |
| zu       | ?       | ?            |

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

AllenAI are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset.

### Citation Information

```
@article{2019t5,
    author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
    title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
    journal = {arXiv e-prints},
    year = {2019},
    archivePrefix = {arXiv},
    eprint = {1910.10683},
}
```

### Contributions

Thanks to [@dirkgr](https://github.com/dirkgr) and [@lhoestq](https://github.com/lhoestq) for adding this dataset.
