Process 0 very slow when using num_procs with map to tokenizer
{'login': 'ksjae', 'id': 17930170, 'node_id': 'MDQ6VXNlcjE3OTMwMTcw', 'avatar_url': 'https://avatars.githubusercontent.com/u/17930170?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/ksjae', 'html_url': 'https://github.com/ksjae', 'followers_url': 'https://api.github.com/users/ksjae/followers', 'following_url': 'https://api.github.com/users/ksjae/following{/other_user}', 'gists_url': 'https://api.github.com/users/ksjae/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/ksjae/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/ksjae/subscriptions', 'organizations_url': 'https://api.github.com/users/ksjae/orgs', 'repos_url': 'https://api.github.com/users/ksjae/repos', 'events_url': 'https://api.github.com/users/ksjae/events{/privacy}', 'received_events_url': 'https://api.github.com/users/ksjae/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Hi ! Thanks for reporting.\r\nIs the distribution of text length of your data evenly distributed across your dataset ? I mean, could it be because the examples in the first part of your dataset are slower to process ?\r\nAlso could how many CPUs can you use for multiprocessing ?\r\n```python\r\nimport multiprocessing\r\nprint(multiprocessing.cpu_count())\r\n```\r\nWhich tokenizer are you using ?'
 "Using pre trained HF tokenizer. The result is the same with tokenizer multiprocessing off and on.\r\nI have (absolutely) no idea about the distribution, but since this issue occurs on all of my datasets(regardless of files), I don't think distribution is the problems.\r\n\r\nI can use up to 16 cores."
 "Ok weird, I don't manage to reproduce this issue on my side.\r\nDoes it happen even with `num_proc=2` for example ?\r\nAlso could you provide more details about your OS and the versions of tokenizers/datasets/multiprocess that you're using ?"
 'Yes, I can confirm it also happens with ```num_proc=2```.\r\n```\r\ntokenizers               0.9.2\r\ndatasets                 1.1.2\r\nmultiprocess             0.70.10\r\n```\r\n```\r\nLinux nipa2020-0629 4.4.0-178-generic #208-Ubuntu SMP Sun Apr 5 23:45:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux\r\n```'
 "I can't reproduce on my side unfortunately with the same versions.\r\n\r\nDo you have issues when doing multiprocessing with python ?\r\n```python\r\nfrom tqdm.auto import tqdm\r\nfrom multiprocess import Pool, RLock\r\n\r\ndef process_data(shard):\r\n    # implement\r\n\r\nnum_proc = 8\r\nshards = [] # implement, this must be a list of size num_proc\r\n\r\nwith Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:\r\n    results = [pool.apply_async(process_data, shard=shard) for shard in shards]\r\n    transformed_shards = [r.get() for r in results]\r\n```"
 "Nah, I'll just wait a few hours. Thank you for helping, though."]
<img width="721" alt="image" src="https://user-images.githubusercontent.com/17930170/97066109-776d0d00-15ed-11eb-8bba-bb4d2e0fcc33.png">
The code I am using is
```

        dataset = load_dataset("text", data_files=[file_path], split='train')
        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                                truncation=True, max_length=args.block_size), num_proc=8)
        dataset.set_format(type='torch', columns=['input_ids'])
        dataset.save_to_disk(file_path+'.arrow')
```

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/758/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}