region:us
# WMT 2014 German-English Translation Dataset

This dataset was built with the fairseq's processing script, which can be original
found [here](https://github.com/facebookresearch/fairseq/blob/main/examples/translation/prepare-wmt14en2de.sh)

You can create this dataset by simply run:

```commandline
git clone https://huggingface.co/datasets/shijli/wmt14-deen
cd wmt14-deen/data
bash prepare-wmt14.sh
```

`binarized.dist.de-en.zip` and `binarized.dist.en-de.zip` are distilled datasets generated by a transformer base model.
It can be built by running:

```commandline
bash prepare-wmt14-distill.sh /path/to/fairseq/model source-lang target-lang
```

To build this dataset, you need to create `binarized.zip` first. Note that the distilled dataset only uses
model-generated
target sentences, which means that different translation directions result in different datasets. Therefore, you need to
specify `source-lang` and `target-lang` explicitly. Also, you need to replace `/path/to/fairseq/model` with the path of
your pretrained model.