region:us
# IWSLT 2014 German-English Translation Dataset w/ further processing

This dataset was built with the fairseq's processing script, which can be originally
found [here](https://github.com/facebookresearch/fairseq/blob/main/examples/translation/prepare-iwslt14.sh)

`iwslt14.tokenized.de-en.zip` and `binarized.zip` can be built by running:

```
git clone https://huggingface.co/datasets/shijli/iwslt14-deen
cd iwslt14-deen/data
bash prepare-iwslt14.sh
```

`binarized.dist.de-en.zip` is a distilled dataset generated by a transformer base model. It can be built by running:

```
bash prepare-iwslt14-distill.sh /path/to/fairseq/model source-lang target-lang
```

To build this dataset, you need to create `binarized.zip` first. Note that the distilled dataset only uses model-generated
target sentences, which means that different translation directions result in different datasets. Therefore, you need to
specify `source-lang` and `target-lang` explicitly. Also, you need to replace `/path/to/fairseq/model` with the path of
your pretrained model.