Add multiprocessing
{'login': 'lhoestq', 'id': 42851186, 'node_id': 'MDQ6VXNlcjQyODUxMTg2', 'avatar_url': 'https://avatars.githubusercontent.com/u/42851186?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lhoestq', 'html_url': 'https://github.com/lhoestq', 'followers_url': 'https://api.github.com/users/lhoestq/followers', 'following_url': 'https://api.github.com/users/lhoestq/following{/other_user}', 'gists_url': 'https://api.github.com/users/lhoestq/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lhoestq/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lhoestq/subscriptions', 'organizations_url': 'https://api.github.com/users/lhoestq/orgs', 'repos_url': 'https://api.github.com/users/lhoestq/repos', 'events_url': 'https://api.github.com/users/lhoestq/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lhoestq/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Logging looks like\r\n\r\n```\r\nDone writing 21900 indices in 3854400 bytes .\r\nProcess #0 will write at playground/tmp_00000_of_00004.arrow\r\nDone writing 21900 indices in 3854400 bytes .\r\nProcess #1 will write at playground/tmp_00001_of_00004.arrow\r\nDone writing 21900 indices in 3854400 bytes .\r\nProcess #2 will write at playground/tmp_00002_of_00004.arrow\r\nDone writing 21899 indices in 3854224 bytes .\r\nProcess #3 will write at playground/tmp_00003_of_00004.arrow\r\nSpawning 4 processes\r\n#3: 100%|████████████████████████████████████████████████| 21899/21899 [00:02<00:00, 8027.41ex/s]\r\n#0: 100%|████████████████████████████████████████████████| 21900/21900 [00:02<00:00, 7982.87ex/s]\r\n#1: 100%|████████████████████████████████████████████████| 21900/21900 [00:02<00:00, 7923.89ex/s]\r\n#2: 100%|████████████████████████████████████████████████| 21900/21900 [00:02<00:00, 7920.04ex/s]\r\nConcatenating 4 shards from multiprocessing\r\n```'
 'I added tests and improved logging.\r\nBoth `map` and `filter` support multiprocessing'
 'A bit strange that the benchmarks on map/filter are worth than `master`.\r\n(maybe because they are not done on the same machine)'
 'The benchmark also got worse in other PRs (see [here](https://github.com/huggingface/nlp/pull/550#commitcomment-41931609) for example, where we have 16sec for `map fast-tokenizer batched` and 18 sec for `map identity`)'
 'Hi,\r\n\r\nwhen I use the multiprocessing in ```.map```:\r\n```\r\ndataset = load_dataset("text", data_files=file_path, split="train")\r\ndataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,\r\n                                      truncation=True, max_length=args.block_size), batched=True, num_proc=16)\r\ndataset.set_format(type=\'torch\', columns=[\'input_ids\'])\r\n```\r\nI get the following error:\r\n```\r\nTraceback (most recent call last):\r\n  File "src/run.py", line 373, in <module>\r\n    main()\r\n  File "src/run.py", line 295, in main\r\n    get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None\r\n  File "src/run.py", line 153, in get_dataset\r\n    dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,\r\n  File "/root/miniconda3/envs/py3.8/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1287, in map\r\n    transformed_shards = [r.get() for r in results]\r\n  File "/root/miniconda3/envs/py3.8/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1287, in <listcomp>\r\n    transformed_shards = [r.get() for r in results]\r\n  File "/root/miniconda3/envs/py3.8/lib/python3.8/multiprocessing/pool.py", line 771, in get\r\n    raise self._value\r\n    put(task)\r\n  File "/root/miniconda3/envs/py3.8/lib/python3.8/multiprocessing/connection.py", line 206, in send\r\n    self._send_bytes(_ForkingPickler.dumps(obj))\r\n  File "/root/miniconda3/envs/py3.8/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps\r\n    cls(buf, protocol).dump(obj)\r\nAttributeError: Can\'t pickle local object \'get_dataset.<locals>.<lambda>\'\r\n```\r\nI think you should use [pathos](https://github.com/uqfoundation/pathos) to pickle the lambda function and some others!\r\nI change the 30 line of src/datasets/arrow_dataset.py as following:\r\n```\r\n# 30 line: from multiprocessing import Pool, RLock\r\nimport pathos\r\nfrom pathos.multiprocessing import Pool\r\nfrom multiprocessing import RLock\r\n```\r\nand it works!'
 "That's very cool indeed !\r\nShall we condiser adding this dependency @thomwolf ?"
 "We already use `dill` so that's definitely a very interesting option indeed!"
 'it gets stuck on debian 9 when num_proc > 1\r\n'
 "Are you using a tokenizer ?\r\nDid you try to set `TOKENIZERS_PARALLELISM=false` ?\r\n\r\nFeel free to discuss it in #620 , we're discussing this issue"
 'I set `TOKENIZERS_PARALLELISM=false`. Just the warning went away. The program was still stuck\r\n']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/552', 'html_url': 'https://github.com/huggingface/datasets/pull/552', 'diff_url': 'https://github.com/huggingface/datasets/pull/552.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/552.patch', 'merged_at': datetime.datetime(2020, 9, 2, 10, 1, 25)}
Adding multiprocessing to `.map`

It works in 3 steps:
- shard the dataset in `num_proc` shards
- spawn one process per shard and call `map` on them
- concatenate the resulting datasets

Example of usage:

```python
from nlp import load_dataset

dataset = load_dataset("squad", split="train")

def function(x):
    return {"lowered": x.lower()}

processed = d.map(
    function,
    input_columns=["context"],
    num_proc=4,
    cache_file_name="playground/tmp.arrow",
    load_from_cache_file=False
)
```

Here it writes 4 files  depending on the process rank:
- `playground/tmp_00000_of_00004.arrow`
- `playground/tmp_00001_of_00004.arrow`
- `playground/tmp_00002_of_00004.arrow`
- `playground/tmp_00003_of_00004.arrow`

The suffix format can be specified by the user.

If the `cache_file_name` is not specified, it writes into separated files depending on the fingerprint, as usual.

I still need to:
- write tests for this
- try to improve the logging (currently it shows 4 progress bars, but if one finishes before the others, then the following messages are written over the progress bars)

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/552/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}