Multiprocessing windows error
{'login': 'flozi00', 'id': 47894090, 'node_id': 'MDQ6VXNlcjQ3ODk0MDkw', 'avatar_url': 'https://avatars.githubusercontent.com/u/47894090?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/flozi00', 'html_url': 'https://github.com/flozi00', 'followers_url': 'https://api.github.com/users/flozi00/followers', 'following_url': 'https://api.github.com/users/flozi00/following{/other_user}', 'gists_url': 'https://api.github.com/users/flozi00/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/flozi00/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/flozi00/subscriptions', 'organizations_url': 'https://api.github.com/users/flozi00/orgs', 'repos_url': 'https://api.github.com/users/flozi00/repos', 'events_url': 'https://api.github.com/users/flozi00/events{/privacy}', 'received_events_url': 'https://api.github.com/users/flozi00/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Hi ! Thanks for reporting.\r\nThis looks like a bug, could you try to provide a minimal code example that reproduces the issue ? This would be very helpful !\r\n\r\nOtherwise I can try to run the wav2vec2 code above on my side but probably not this week..'
 "```\r\nfrom datasets import load_dataset\r\n\r\ndataset = load_dataset('glue', 'mrpc', split='train')\r\n\r\n\r\nupdated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)\r\n\r\n```"
 '\r\n\r\n\r\n\r\n\r\nI was able to copy some of the shell \r\nThis is repeating every half second\r\nWin 10, Anaconda with python 3.8, datasets installed from main branche\r\n```\r\n\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 287, in _fixup_main_from_path\r\n    _check_not_importing_main()\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 116, in spawn_main\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 134, in _check_not_importing_main\r\n    main_content = runpy.run_path(main_path,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 265, in run_path\r\n    exitcode = _main(fd, parent_sentinel)\r\n    raise RuntimeError(\'\'\'\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 125, in _main\r\nRuntimeError:\r\n        An attempt has been made to start a new process before the\r\n        current process has finished its bootstrapping phase.\r\n\r\n        This probably means that you are not using fork to start your\r\n        child processes and you have forgotten to use the proper idiom\r\n        in the main module:\r\n\r\n            if __name__ == \'__main__\':\r\n                freeze_support()\r\n                ...\r\n\r\n        The "freeze_support()" line can be omitted if the program\r\n        is not going to be frozen to produce an executable.    return _run_module_code(code, init_globals, run_name,\r\n    prepare(preparation_data)\r\n\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 97, in _run_module_code\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 236, in prepare\r\n    _run_code(code, mod_globals, init_globals,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 87, in _run_code\r\n    _fixup_main_from_path(data[\'init_main_from_path\'])\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 287, in _fixup_main_from_path\r\n    exec(code, run_globals)\r\n  File "F:\\Codes\\Python Apps\\asr\\test.py", line 6, in <module>\r\n    updated_dataset = dataset.map(lambda example: {\'sentence1\': \'My sentence: \' + example[\'sentence1\']}, num_proc=4)\r\n    main_content = runpy.run_path(main_path,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\datasets\\arrow_dataset.py", line 1370, in map\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 265, in run_path\r\n    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\context.py", line 119, in Pool\r\n    return _run_module_code(code, init_globals, run_name,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 97, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n    return Pool(processes, initializer, initargs, maxtasksperchild,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 87, in _run_code\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\pool.py", line 212, in __init__\r\n    exec(code, run_globals)\r\n  File "F:\\Codes\\Python Apps\\asr\\test.py", line 6, in <module>\r\n    self._repopulate_pool()\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\pool.py", line 303, in _repopulate_pool\r\n    updated_dataset = dataset.map(lambda example: {\'sentence1\': \'My sentence: \' + example[\'sentence1\']}, num_proc=4)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\datasets\\arrow_dataset.py", line 1370, in map\r\n    return self._repopulate_pool_static(self._ctx, self.Process,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\pool.py", line 326, in _repopulate_pool_static\r\n    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\context.py", line 119, in Pool\r\n    w.start()\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\process.py", line 121, in start\r\n    return Pool(processes, initializer, initargs, maxtasksperchild,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\pool.py", line 212, in __init__\r\n    self._popen = self._Popen(self)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\context.py", line 327, in _Popen\r\n    self._repopulate_pool()\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\pool.py", line 303, in _repopulate_pool\r\n    return Popen(process_obj)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\popen_spawn_win32.py", line 45, in __init__\r\n    return self._repopulate_pool_static(self._ctx, self.Process,\r\n    prep_data = spawn.get_preparation_data(process_obj._name)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\pool.py", line 326, in _repopulate_pool_static\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 154, in get_preparation_data\r\n    _check_not_importing_main()\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 134, in _check_not_importing_main\r\n    w.start()\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\process.py", line 121, in start\r\n    raise RuntimeError(\'\'\'\r\nRuntimeError:\r\n        An attempt has been made to start a new process before the\r\n        current process has finished its bootstrapping phase.\r\n\r\n        This probably means that you are not using fork to start your\r\n        child processes and you have forgotten to use the proper idiom\r\n        in the main module:\r\n\r\n            if __name__ == \'__main__\':\r\n                freeze_support()\r\n                ...\r\n```'
 "Thanks this is really helpful !\r\nI'll try to reproduce on my side and come back to you"
 "if __name__ == '__main__':\r\n\r\n\r\nThis line before calling the map function stops the error but the script still repeats endless"
 "Indeed you needed `if __name__ == '__main__'` since accoding to [this stackoverflow post](https://stackoverflow.com/a/18205006):\r\n\r\n> On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == '__main__': guard in the main module to avoid creating subprocesses recursively.\r\n\r\nRegarding the hanging issue, can you try to update `dill` and `multiprocess` ?"
 "It's already on the newest version"
 '```\r\nTraceback (most recent call last):\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\shutil.py", line 791, in move\r\n    os.rename(src, real_dst)\r\nFileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: \'D:\\\\huggingfacecache\\\\common_voice\\\\de\\\\6.1.0\\\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\\\tmpx9fl_jg8\' -> \'D:\\\\huggingfacecache\\\\common_voice\\\\de\\\\6.1.0\\\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\\\cache-9b4f203a63742dfc.arrow\'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File "<string>", line 1, in <module>\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 116, in spawn_main\r\n    exitcode = _main(fd, parent_sentinel)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 125, in _main\r\n    prepare(preparation_data)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 236, in prepare\r\n    _fixup_main_from_path(data[\'init_main_from_path\'])\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\multiprocess\\spawn.py", line 287, in _fixup_main_from_path\r\n    main_content = runpy.run_path(main_path,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 265, in run_path\r\n    return _run_module_code(code, init_globals, run_name,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 97, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\runpy.py", line 87, in _run_code\r\n    exec(code, run_globals)\r\n  File "F:\\Codes\\Python Apps\\asr\\cvtrain.py", line 243, in <module>\r\n    common_voice_train = common_voice_train.map(remove_special_characters, remove_columns=["sentence"])\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\datasets\\arrow_dataset.py", line 1339, in map\r\n    return self._map_single(\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\datasets\\arrow_dataset.py", line 203, in wrapper\r\n    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\datasets\\fingerprint.py", line 337, in wrapper\r\n    out = func(self, *args, **kwargs)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\site-packages\\datasets\\arrow_dataset.py", line 1646, in _map_single\r\n    shutil.move(tmp_file.name, cache_file_name)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\shutil.py", line 805, in move\r\n    copy_function(src, real_dst)\r\n  File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\shutil.py", line 435, in copy2\r\n    copyfile(src, dst, follow_symlinks=follow_symlinks)\r\n  0%|                                                                                                                                                                                                 | 0/27771 [00:00<?, ?ex/s] \r\n File "C:\\Users\\flozi\\anaconda3\\envs\\wav2vec\\lib\\shutil.py", line 264, in copyfile\r\n    with open(src, \'rb\') as fsrc, open(dst, \'wb\') as fdst:\r\nOSError: [Errno 22] Invalid argument: \'D:\\\\huggingfacecache\\\\common_voice\\\\de\\\\6.1.0\\\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\\\cache-9b4f203a63742dfc.arrow\'\r\n```\r\n\r\nI was adding freeze support before calling the mapping function like this\r\nif __name__ == \'__main__\':\r\n    freeze_support()\r\n   dataset.map(....)'
 "Usually OSError of an arrow file on windows means that the file is currently opened as a dataset object, so you can't overwrite it until the dataset object falls out of scope.\r\nCan you make sure that there's no dataset object that loaded the `cache-9b4f203a63742dfc.arrow` file ?"
 "Now I understand\r\nThe error occures because the script got restarted in another thread, so the object is already loaded.\r\nStill don't have an idea why a new thread starts the whole script again"]
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2067/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}