task_categories:text2text-generation task_categories:translation annotations_creators:found language_creators:found multilinguality:multilingual source_datasets:original language:af language:am language:ar language:ast language:az language:be language:bg language:bn language:br language:ca language:ceb language:cs language:cy language:da language:de language:el language:en language:eo language:es language:et language:eu language:fa language:fi language:fr language:fy language:ga language:gd language:gl language:ha language:he language:hi language:hr language:hu language:hy language:id language:ig language:ilo language:is language:it language:ja language:jv language:ka language:kk language:km language:ko language:la language:lb language:lg language:lt language:lv language:mg language:mk language:ml language:mr language:ms language:my language:ne language:nl language:no language:oc language:om language:or language:pl language:pt language:ro language:ru language:sd language:si language:sk language:sl language:so language:sq language:sr language:su language:sv language:sw language:ta language:tl language:tr language:tt language:uk language:ur language:uz language:vi language:wo language:xh language:yi language:yo language:zh language:zu language:se license:unknown conditional-text-generation arxiv:1911.04944 arxiv:1911.00359 arxiv:2010.11125 region:us
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB

We show that margin-based bitext mining in LASER's multilingual sentence space can be applied to
monolingual corpora of billions of sentences to produce high quality aligned translation data.
We use thirty-two snapshots of a curated common crawl corpus [1] totaling 69 billion unique sentences.
Using one unified approach for 80 languages, we were able to mine 10.8 billion parallel sentences,
out of which only 2.9 billion are aligned with English.

IMPORTANT: Please cite reference [2][3] if you use this data.

[1] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Jouli
    and Edouard Grave, CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

[2] Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave and Armand Joulin,
    CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB

[3] Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines,
    Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky,
    Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin.
    Beyond English-Centric Multilingual Machine Translation
    
90 languages, 1,197 bitexts
total number of files: 90
total number of tokens: 112.14G
total number of sentence fragments: 7.37G
 Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Jouli and Edouard Grave, CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
---
annotations_creators:
- found
language_creators:
- found
language:
- af
- am
- ar
- ast
- az
- be
- bg
- bn
- br
- ca
- ceb
- cs
- cy
- da
- de
- el
- en
- eo
- es
- et
- eu
- fa
- fi
- fr
- fy
- ga
- gd
- gl
- ha
- he
- hi
- hr
- hu
- hy
- id
- ig
- ilo
- is
- it
- ja
- jv
- ka
- kk
- km
- ko
- la
- lb
- lg
- lt
- lv
- mg
- mk
- ml
- mr
- ms
- my
- ne
- nl
- 'no'
- oc
- om
- or
- pl
- pt
- ro
- ru
- sd
- si
- sk
- sl
- so
- sq
- sr
- su
- sv
- sw
- ta
- tl
- tr
- tt
- uk
- ur
- uz
- vi
- wo
- xh
- yi
- yo
- zh
- zu
- se
license:
- unknown
multilinguality:
- multilingual
size_categories:
  en-nl:
  - n<110M
  en-af:
  - n<9M
  en-lt:
  - <24M
source_datasets:
- original
task_categories:
- text2text-generation
- translation
task_ids: []
paperswithcode_id: ccmatrix
pretty_name: CCMatrixV1
tags:
- conditional-text-generation
---

# Dataset Card for CCMatrix v1

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description
- **Homepage:** https://opus.nlpl.eu/CCMatrix.php
- **Repository:** None
- **Paper:** https://arxiv.org/abs/1911.04944
### Dataset Summary

This corpus has been extracted from web crawls using the margin-based bitext mining techniques described at https://github.com/facebookresearch/LASER/tree/master/tasks/CCMatrix.

* 90 languages, 1,197 bitexts
* total number of files: 90
* total number of tokens: 112.14G
* total number of sentence fragments: 7.37G

### Supported Tasks and Leaderboards
[More Information Needed]

### Languages

Configs are generated for all language pairs in both directions.
You can find the valid pairs in Homepage section of Dataset Description: https://opus.nlpl.eu/CCMatrix.php
E.g.

```
from datasets import load_dataset
dataset = load_dataset("yhavinga/ccmatrix", "en-nl", streaming=True)
```

This will open the `en-nl` dataset in streaming mode. Without streaming, download and prepare will take tens of minutes.
You can inspect elements with:

```
print(next(iter(dataset['train'])))
{'id': 0, 'score': 1.2499677, 'translation': {'en': 'They come from all parts of Egypt, just like they will at the day of His coming.', 'nl': 'Zij kwamen uit alle delen van Egypte, evenals zij op de dag van Zijn komst zullen doen.'}}
```

## Dataset Structure
### Data Instances
For example:

```json
{
        "id": 1,
        "score": 1.2498379,
        "translation": {
            "nl": "En we moeten elke waarheid vals noemen die niet minstens door een lach vergezeld ging.”",
            "en": "And we should call every truth false which was not accompanied by at least one laugh.”"
        }
    }
```

### Data Fields
Each example contains an integer id starting with 0, a score, and a translation dictionary with the language 1 and
language 2 texts.

### Data Splits
Only a `train` split is provided.

## Dataset Creation
### Curation Rationale
[More Information Needed]
### Source Data
[More Information Needed]
#### Initial Data Collection and Normalization
[More Information Needed]
#### Who are the source language producers?
[More Information Needed]
### Annotations
[More Information Needed]
#### Annotation process
[More Information Needed]
#### Who are the annotators?
[More Information Needed]
### Personal and Sensitive Information
[More Information Needed]
## Considerations for Using the Data
### Social Impact of Dataset
[More Information Needed]
### Discussion of Biases
[More Information Needed]
### Other Known Limitations
[More Information Needed]
## Additional Information
### Dataset Curators
[More Information Needed]
### Licensing Information
[More Information Needed]
### Citation Information

IMPORTANT: Please cite reference [2][3] if you use this data.

1. **[CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://arxiv.org/abs/1911.00359)**
   by *Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Jouli
    and Edouard Grave*.
2. **[CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://arxiv.org/abs/1911.04944)** by *Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave and Armand Joulin*.
3. **[Beyond English-Centric Multilingual Machine Translation](https://arxiv.org/abs/2010.11125)** by *Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines,
    Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky,
    Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin.*

This HuggingFace CCMatrix dataset is a wrapper around the service and files prepared and hosted by OPUS:

* **[Parallel Data, Tools and Interfaces in OPUS](https://www.aclweb.org/anthology/L12-1246/)** by *Jörg Tiedemann*.

### Contributions
