task_categories:translation annotations_creators:no-annotation language_creators:found multilinguality:translation size_categories:10M<n<100M source_datasets:original language:bs language:bg language:en language:is language:hr language:cnr language:mk language:mt language:sl language:sr language:sq language:tr license:cc0-1.0 region:us
The MaCoCu parallel dataset is an English-centric collection of 11
parallel corpora including the following languages: Albanian,
Bulgarian, Bosnian, Croatian, Icelandic, Macedonian, Maltese,
Montenegrin, Serbian, Slovenian, and Turkish. These corpora have
been automatically crawled from national and generic top-level
domains (for example, ".hr" for croatian, or ".is" for icelandic);
then, a parallel curation pipeline has been applied to produce
the final data (see https://github.com/bitextor/bitextor).
@inproceedings{banon2022macocu,
  title={MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages},
  author={Ban{\'o}n, Marta and Espla-Gomis, Miquel and Forcada, Mikel L and Garc{\'\i}a-Romero, Cristian and Kuzman, Taja and Ljube{\v{s}}i{\'c}, Nikola and van Noord, Rik and Sempere, Leopoldo Pla and Ram{\'\i}rez-S{\'a}nchez, Gema and Rupnik, Peter and others},
  booktitle={23rd Annual Conference of the European Association for Machine Translation, EAMT 2022},
  pages={303--304},
  year={2022},
  organization={European Association for Machine Translation}
}
---
annotations_creators:
- no-annotation
language_creators:
- found
language:
- bs
- bg
- en
- is 
- hr
- cnr
- mk
- mt
- sl
- sr
- sq
- tr
license:
- cc0-1.0
multilinguality:
- translation
pretty_name: MaCoCu_parallel
size_categories:
- 10M<n<100M
source_datasets:
- original
task_categories:
- translation
task_ids: []
dataset_info:
- config_name: enis
  features:
  - name: translation
    dtype:
      translation:
        languages:
        - is
        - en
  splits:
  - name: train
    num_bytes: 133883139
    num_examples: 546172
  download_size: 133883139
  dataset_size: 133883139
- config_name: enbg
  features:
  - name: translation
    dtype:
      translation:
        languages:
        - bg
        - en
  splits:
  - name: train
    num_bytes: 133883139
    num_examples: 546172
  download_size: 133883139
  dataset_size: 133883139
---
license: cc0-1.0
---
### Dataset Summary
The collection of MaCoCu parallel corpora have been crawled and consist of pairs of source and target segments (one or several sentences) and additional metadata. The following metadata is included:
- "src_url" and "trg_url": source and target document URL;
- "src_text" and "trg_text": text in non-English language and in English Language;
- "bleualign_score": similarity score as provided by the sentence alignment tool Bleualign (value between 0 and 1);
- "src_deferred_hash" and "trg_deferred_hash": hash identifier for the corresponding segment;
- "src_paragraph_id" and "trg_paragraph_id": identifier of the paragraph where the segment appears in the original document;
- "src_doc_title" and "trg_doc_title": title of the documents from which segments where obtained;
- "src_crawl_date" and "trg_crawl_date": date and time when source and target documents where donwoaded;
- "src_file_type" and "trg_file_type": type of the original documents (usually HTML format);
- "src_boilerplate" and "trg_boilerplate": are source or target segments boilerplates?
- "bifixer_hash": hash identifier for the segment pair;
- "bifixer_score": score that indicates how likely are segments to be correct in their corresponding language;
- "bicleaner_ai_score": score that indicates how likely are segments to be parallel;
- "biroamer_entities_detected": do any of the segments contain personal information?
- "dsi": a DSI class (“dsi”): information whether the segment is connected to any of Digital Service Infrastructure (DSI) classes (e.g., cybersecurity, e-health, e-justice, open-data-portal), defined by the Connecting Europe Facility (https://github.com/RikVN/DSI);
- "translation_direction": translation direction and machine translation identification ("translation-direction"): the source segment in each segment pair was identified by using a probabilistic model (https://github.com/RikVN/TranslationDirection), which also determines if the translation has been produced by a machine-translation system;
- "en_document_level_variant": the language variant of English (British or American, using a lexicon-based English variety classifier - https://pypi.org/project/abclf/) was identified on document and domain level; 
- "domain_en": name of the web domain for the English document;
- "en_domain_level_variant": language variant for English at the level of the web domain.

To load a language pair just indicate the dataset and the pair of languages with English first

```python
dataset = load_dataset("MaCoCu/parallel_data", "en-is")
```
