task_categories:translation size_categories:10M<n<100M language:en language:fr license:odbl webpages parallel parallel data region:us
---
language:
- en
- fr
license: odbl
size_categories:
- 10M<n<100M
task_categories:
- translation
tags:
- webpages
- parallel
- parallel data
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
dataset_info:
  features:
  - name: en
    dtype: string
  - name: fr
    dtype: string
  splits:
  - name: train
    num_bytes: 6811772380
    num_examples: 17161263
  download_size: 640497280
  dataset_size: 6811772380
---

# English French Webpages Scraped Translated

### Dataset Summary
French/English parallel texts for training translation models. Over 17.1 million sentences in French and English. Dataset created by Chris Callison-Burch, who crawled millions of web pages and then used a set of simple heuristics to transform French URLs onto English URLs, and assumed that these documents are translations of each other. This is the main dataset of Workshop on Statistical Machine Translation (WML) 2015 Dataset that can be used for Machine Translation and Language Models. Refer to the paper here: http://www.statmt.org/wmt15/pdf/WMT01.pdf

### Post-process
This dataset has been post-processed to remove all duplicates, empty fields and phrases containing less than 5 words.

### Original Dataset Citation

```
@InProceedings{bojar-EtAl:2015:WMT,
  author    = {Bojar, Ond\v{r}ej  and  Chatterjee, Rajen  and  Federmann, Christian  and  Haddow, Barry  and  Huck, Matthias  and  Hokamp, Chris  and  Koehn, Philipp  and  Logacheva, Varvara  and  Monz, Christof  and  Negri, Matteo  and  Post, Matt  and  Scarton, Carolina  and  Specia, Lucia  and  Turchi, Marco},
  title     = {Findings of the 2015 Workshop on Statistical Machine Translation},
  booktitle = {Proceedings of the Tenth Workshop on Statistical Machine Translation},
  month     = {September},
  year      = {2015},
  address   = {Lisbon, Portugal},
  publisher = {Association for Computational Linguistics},
  pages     = {1--46},
  url       = {http://aclweb.org/anthology/W15-3001}
}
```