Use multiprocess from pathos for multiprocessing
{'login': 'lhoestq', 'id': 42851186, 'node_id': 'MDQ6VXNlcjQyODUxMTg2', 'avatar_url': 'https://avatars.githubusercontent.com/u/42851186?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lhoestq', 'html_url': 'https://github.com/lhoestq', 'followers_url': 'https://api.github.com/users/lhoestq/followers', 'following_url': 'https://api.github.com/users/lhoestq/following{/other_user}', 'gists_url': 'https://api.github.com/users/lhoestq/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lhoestq/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lhoestq/subscriptions', 'organizations_url': 'https://api.github.com/users/lhoestq/orgs', 'repos_url': 'https://api.github.com/users/lhoestq/repos', 'events_url': 'https://api.github.com/users/lhoestq/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lhoestq/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["We can just install multiprocess actually, I'll change that"
 'Just an FYI: I remember that I wanted to try pathos a couple of years back and I ran into issues considering cross-platform; the code would just break on Windows. If I can verify this PR by running CPU tests on Windows, let me know!'
 "That's good to know thanks\r\nI guess we can just wait for #644 to be merged first. I'm working on fixing the tests for windows"
 'Looks like all the CI jobs on windows passed !\r\nI also tested locally on my windows and it works great :) \r\n\r\nI think this is ready to merge, let me know if you have any remarks @thomwolf @BramVanroy ']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/656', 'html_url': 'https://github.com/huggingface/datasets/pull/656', 'diff_url': 'https://github.com/huggingface/datasets/pull/656.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/656.patch', 'merged_at': datetime.datetime(2020, 9, 28, 14, 45, 39)}
[Multiprocess](https://github.com/uqfoundation/multiprocess) (from the [pathos](https://github.com/uqfoundation/pathos) project) allows to use lambda functions in multiprocessed map.
It was suggested to use it by @kandorm.

We're already using dill which is its only dependency.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/656/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}