Support multi-worker with streaming dataset (IterableDataset).
{'login': 'cccntu', 'id': 31893406, 'node_id': 'MDQ6VXNlcjMxODkzNDA2', 'avatar_url': 'https://avatars.githubusercontent.com/u/31893406?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/cccntu', 'html_url': 'https://github.com/cccntu', 'followers_url': 'https://api.github.com/users/cccntu/followers', 'following_url': 'https://api.github.com/users/cccntu/following{/other_user}', 'gists_url': 'https://api.github.com/users/cccntu/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/cccntu/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/cccntu/subscriptions', 'organizations_url': 'https://api.github.com/users/cccntu/orgs', 'repos_url': 'https://api.github.com/users/cccntu/repos', 'events_url': 'https://api.github.com/users/cccntu/events{/privacy}', 'received_events_url': 'https://api.github.com/users/cccntu/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing. 

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step
**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2642/reactions', 'total_count': 1, '+1': 1, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}