[FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter
{'login': 'timothyjlaurent', 'id': 2000204, 'node_id': 'MDQ6VXNlcjIwMDAyMDQ=', 'avatar_url': 'https://avatars.githubusercontent.com/u/2000204?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/timothyjlaurent', 'html_url': 'https://github.com/timothyjlaurent', 'followers_url': 'https://api.github.com/users/timothyjlaurent/followers', 'following_url': 'https://api.github.com/users/timothyjlaurent/following{/other_user}', 'gists_url': 'https://api.github.com/users/timothyjlaurent/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/timothyjlaurent/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/timothyjlaurent/subscriptions', 'organizations_url': 'https://api.github.com/users/timothyjlaurent/orgs', 'repos_url': 'https://api.github.com/users/timothyjlaurent/repos', 'events_url': 'https://api.github.com/users/timothyjlaurent/events{/privacy}', 'received_events_url': 'https://api.github.com/users/timothyjlaurent/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
["Yes that's definitely something we plan to add ^^"
 'Yes, that would be nice. We could take a look at what tensorflow `tf.data` does under the hood for instance.'
 'So `tf.data.Dataset.map()` returns a `ParallelMapDataset` if `num_parallel_calls is not None` [link](https://github.com/tensorflow/tensorflow/blob/2b96f3662bd776e277f86997659e61046b56c315/tensorflow/python/data/ops/dataset_ops.py#L1623).\r\n\r\nThere, `num_parallel_calls` is turned into a tensor and and fed to `gen_dataset_ops.parallel_map_dataset` where it looks like tensorflow takes over.\r\n\r\nWe could start with something simple like a thread or process pool that `imap`s over some shards.\r\n '
 'Multiprocessing was added in #552 . You can set the number of processes with `.map(..., num_proc=...)`. It also works for `filter`\r\n\r\nClosing this one, but feel free to reo-open if you have other questions'
 '@lhoestq Great feature implemented! Do you have plans to add it to official tutorials [Processing data in a Dataset](https://huggingface.co/docs/datasets/processing.html?highlight=save#augmenting-the-dataset)? It took me sometime to find this parallel processing api.'
 'Thanks for the heads up !\r\n\r\nI just added a paragraph about multiprocessing:\r\nhttps://huggingface.co/docs/datasets/master/processing.html#multiprocessing']
It would be nice to be able to speed up `dataset.map` or `dataset.filter`. Perhaps this is as easy as sharding the dataset sending each shard to a process/thread/dask pool and using the new `nlp.concatenate_dataset()` function to join them all together?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/426/reactions', 'total_count': 3, '+1': 3, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}