wmt download speed example
{'login': 'sshleifer', 'id': 6045025, 'node_id': 'MDQ6VXNlcjYwNDUwMjU=', 'avatar_url': 'https://avatars.githubusercontent.com/u/6045025?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/sshleifer', 'html_url': 'https://github.com/sshleifer', 'followers_url': 'https://api.github.com/users/sshleifer/followers', 'following_url': 'https://api.github.com/users/sshleifer/following{/other_user}', 'gists_url': 'https://api.github.com/users/sshleifer/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/sshleifer/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/sshleifer/subscriptions', 'organizations_url': 'https://api.github.com/users/sshleifer/orgs', 'repos_url': 'https://api.github.com/users/sshleifer/repos', 'events_url': 'https://api.github.com/users/sshleifer/events{/privacy}', 'received_events_url': 'https://api.github.com/users/sshleifer/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["Thanks for creating the issue :)\r\nThe download link for wmt-en-de raw looks like a mirror. We should use that instead of the current url.\r\nIs this mirror official ?\r\n\r\nAlso it looks like for `ro-en` it tried to download other languages. If we manage to only download the one that is asked it'd be cool\r\n\r\nAlso cc @patrickvonplaten "
 'Mirror is not official.'
 'Shall we host the files ourselves or it is fine to use this mirror in your opinion ?'
 'Should we add an argument in `load_dataset` to override some URL with a custom URL (e.g. mirror) or a local path?\r\n\r\nThis could also be used to provide local files instead of the original files as requested by some users (e.g. when you made a dataset with the same format than SQuAD and what to use it instead of the official dataset files).'
 "@lhoestq I think we should host it ourselves. I'll put the subset of wmt (without preprocessed files) that we need on s3 and post a link over the weekend."
 'Is there a solution yet? The download speed is still too slow. 60-70kbps download for wmt16 and around 100kbps for wmt19. @sshleifer '
 "I'm working on mirror links which will provide high download speed :)\r\nSee https://github.com/huggingface/datasets/issues/1892"]
Continuing from the slack 1.0 roadmap thread w @lhoestq , I realized the slow downloads is only a thing sometimes. Here are a few examples, I suspect there are multiple issues. All commands were run from the same gcp us-central-1f machine.

```
import nlp
nlp.load_dataset('wmt16', 'de-en')
```
Downloads at 49.1 KB/S

Whereas 
```
pip install gdown # download from google drive
!gdown https://drive.google.com/uc?id=1iO7um-HWoNoRKDtw27YUSgyeubn9uXqj
```
Downloads at 127 MB/s. (The file is a copy of wmt-en-de raw).


```
nlp.load_dataset('wmt16', 'ro-en')
```
goes at 27 MB/s, much faster. 

if we wget the same data from s3 is the same download speed, but ¼ the file size:
```
wget https://s3.amazonaws.com/datasets.huggingface.co/translation/wmt_en_ro_packed_200_rand.tgz
```

Finally,
```
nlp.load_dataset('wmt19', 'zh-en')
```
Starts fast, but broken. (duplicate of #493 )



{'url': 'https://api.github.com/repos/huggingface/datasets/issues/525/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}