How to enable `.map()` pre-processing pipelines to support multi-node parallelism?
{'login': 'shangw-nvidia', 'id': 66387198, 'node_id': 'MDQ6VXNlcjY2Mzg3MTk4', 'avatar_url': 'https://avatars.githubusercontent.com/u/66387198?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/shangw-nvidia', 'html_url': 'https://github.com/shangw-nvidia', 'followers_url': 'https://api.github.com/users/shangw-nvidia/followers', 'following_url': 'https://api.github.com/users/shangw-nvidia/following{/other_user}', 'gists_url': 'https://api.github.com/users/shangw-nvidia/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/shangw-nvidia/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/shangw-nvidia/subscriptions', 'organizations_url': 'https://api.github.com/users/shangw-nvidia/orgs', 'repos_url': 'https://api.github.com/users/shangw-nvidia/repos', 'events_url': 'https://api.github.com/users/shangw-nvidia/events{/privacy}', 'received_events_url': 'https://api.github.com/users/shangw-nvidia/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Right now multiprocessing only runs on single node.

However it's probably possible to extend it to support multi nodes. Indeed we're using the `multiprocess` library from the `pathos` project to do multiprocessing in `datasets`, and `pathos` is made to support parallelism on several nodes. More info about pathos [on the pathos repo](https://github.com/uqfoundation/pathos).

If you're familiar with pathos or if you want to give it a try, it could be a nice addition to the library :) Curious to hear if anything on that side changed or if you suggestions to do it changed @lhoestq :)

For our use-case, we are entering the regime where trading a few more instances to save a few days would be nice :) Currently for multi-node setups we're mostly going towards a nice integration with Dask. But I wouldn't exclude exploring `pathos` more at one point
Hi,

Currently, multiprocessing can be enabled for the `.map()` stages on a single node. However, in the case of multi-node training, (since more than one node would be available) I'm wondering if it's possible to extend the parallel processing among nodes, instead of only 1 node running the `.map()` while the other node is waiting for it to finish?

Thanks!
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/842/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}