multiprocessing in dataset map "can only test a child process"
{'login': 'timothyjlaurent', 'id': 2000204, 'node_id': 'MDQ6VXNlcjIwMDAyMDQ=', 'avatar_url': 'https://avatars.githubusercontent.com/u/2000204?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/timothyjlaurent', 'html_url': 'https://github.com/timothyjlaurent', 'followers_url': 'https://api.github.com/users/timothyjlaurent/followers', 'following_url': 'https://api.github.com/users/timothyjlaurent/following{/other_user}', 'gists_url': 'https://api.github.com/users/timothyjlaurent/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/timothyjlaurent/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/timothyjlaurent/subscriptions', 'organizations_url': 'https://api.github.com/users/timothyjlaurent/orgs', 'repos_url': 'https://api.github.com/users/timothyjlaurent/repos', 'events_url': 'https://api.github.com/users/timothyjlaurent/events{/privacy}', 'received_events_url': 'https://api.github.com/users/timothyjlaurent/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["It looks like an issue with wandb/tqdm here.\r\nWe're using the `multiprocess` library instead of the `multiprocessing` builtin python package to support various types of mapping functions. Maybe there's some sort of incompatibility.\r\n\r\nCould you make a minimal script to reproduce or a google colab ?"
 'hi facing the same issue here - \r\n\r\n`AssertionError: Caught AssertionError in DataLoader worker process 0.\r\nOriginal Traceback (most recent call last):\r\n  File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit\r\n    stream.write(msg)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/lib/redirect.py", line 100, in new_write\r\n    cb(name, data)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/wandb_run.py", line 723, in _console_callback\r\n    self._backend.interface.publish_output(name, data)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/interface/interface.py", line 153, in publish_output\r\n    self._publish_output(o)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/interface/interface.py", line 158, in _publish_output\r\n    self._publish(rec)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/interface/interface.py", line 456, in _publish\r\n    if self._process and not self._process.is_alive():\r\n  File "/usr/lib/python3.6/multiprocessing/process.py", line 134, in is_alive\r\n    assert self._parent_pid == os.getpid(), \'can only test a child process\'\r\nAssertionError: can only test a child process\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop\r\n    data = fetcher.fetch(index)\r\n  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch\r\n    data = [self.dataset[idx] for idx in possibly_batched_index]\r\n  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>\r\n    data = [self.dataset[idx] for idx in possibly_batched_index]\r\n  File "<ipython-input-8-a4d9a08d114e>", line 20, in __getitem__\r\n    return_token_type_ids=True\r\n  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 2405, in encode_plus\r\n    **kwargs,\r\n  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 2125, in _get_padding_truncation_strategies\r\n    "Truncation was not explicitly activated but `max_length` is provided a specific value, "\r\n  File "/usr/lib/python3.6/logging/__init__.py", line 1320, in warning\r\n    self._log(WARNING, msg, args, **kwargs)\r\n  File "/usr/lib/python3.6/logging/__init__.py", line 1444, in _log\r\n    self.handle(record)\r\n  File "/usr/lib/python3.6/logging/__init__.py", line 1454, in handle\r\n    self.callHandlers(record)\r\n  File "/usr/lib/python3.6/logging/__init__.py", line 1516, in callHandlers\r\n    hdlr.handle(record)\r\n  File "/usr/lib/python3.6/logging/__init__.py", line 865, in handle\r\n    self.emit(record)\r\n  File "/usr/lib/python3.6/logging/__init__.py", line 1000, in emit\r\n    self.handleError(record)\r\n  File "/usr/lib/python3.6/logging/__init__.py", line 917, in handleError\r\n    sys.stderr.write(\'--- Logging error ---\\n\')\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/lib/redirect.py", line 100, in new_write\r\n    cb(name, data)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/wandb_run.py", line 723, in _console_callback\r\n    self._backend.interface.publish_output(name, data)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/interface/interface.py", line 153, in publish_output\r\n    self._publish_output(o)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/interface/interface.py", line 158, in _publish_output\r\n    self._publish(rec)\r\n  File "/usr/local/lib/python3.6/dist-packages/wandb/sdk/interface/interface.py", line 456, in _publish\r\n    if self._process and not self._process.is_alive():\r\n  File "/usr/lib/python3.6/multiprocessing/process.py", line 134, in is_alive\r\n    assert self._parent_pid == os.getpid(), \'can only test a child process\'\r\nAssertionError: can only test a child process`\r\n'
 'It looks like this warning : \r\n"Truncation was not explicitly activated but max_length is provided a specific value, "\r\nis not handled well by wandb.\r\n\r\nThe error occurs when calling the tokenizer.\r\nMaybe you can try to specify `truncation=True` when calling the tokenizer to remove the warning ?\r\nOtherwise I don\'t know why wandb would fail on a warning. Maybe one of its logging handlers have some issues with the logging of tokenizers. Maybe @n1t0 knows more about this ?'
 'I\'m having a similar issue but when I try to do multiprocessing with the `DataLoader`\r\n\r\nCode to reproduce:\r\n\r\n```\r\nfrom datasets import load_dataset\r\n\r\nbook_corpus = load_dataset(\'bookcorpus\', \'plain_text\', cache_dir=\'/home/ad/Desktop/bookcorpus\', split=\'train[:1%]\')\r\nbook_corpus = book_corpus.map(encode, batched=True, num_proc=20, load_from_cache_file=True, batch_size=5000)\r\nbook_corpus.set_format(type=\'torch\', columns=[\'text\', "input_ids", "attention_mask", "token_type_ids"])\r\n\r\nfrom transformers import DataCollatorForWholeWordMask\r\nfrom transformers import Trainer, TrainingArguments\r\n\r\ndata_collator = DataCollatorForWholeWordMask(\r\n    tokenizer=tokenizer, mlm=True, mlm_probability=0.15)\r\n\r\ntraining_args = TrainingArguments(\r\n    output_dir="./mobile_linear_att_8L_128_128_03layerdrop_shared",\r\n    overwrite_output_dir=True,\r\n    num_train_epochs=1,\r\n    per_device_train_batch_size=64,\r\n    save_steps=50,\r\n    save_total_limit=2,\r\n    logging_first_step=True,\r\n    warmup_steps=100,\r\n    logging_steps=50,\r\n    gradient_accumulation_steps=1,\r\n    fp16=True,\r\n    **dataloader_num_workers=10**,\r\n)\r\n\r\ntrainer = Trainer(\r\n    model=model,\r\n    args=training_args,\r\n    data_collator=data_collator,\r\n    train_dataset=book_corpus,\r\n    tokenizer=tokenizer)\r\n\r\ntrainer.train()\r\n```\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nAssertionError                            Traceback (most recent call last)\r\n<timed eval> in <module>\r\n\r\n~/anaconda3/envs/tfm/lib/python3.6/site-packages/transformers/trainer.py in train(self, model_path, trial)\r\n    869             self.control = self.callback_handler.on_epoch_begin(self.args, self.state, self.control)\r\n    870 \r\n--> 871             for step, inputs in enumerate(epoch_iterator):\r\n    872 \r\n    873                 # Skip past any already trained steps if resuming training\r\n\r\n~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)\r\n    433         if self._sampler_iter is None:\r\n    434             self._reset()\r\n--> 435         data = self._next_data()\r\n    436         self._num_yielded += 1\r\n    437         if self._dataset_kind == _DatasetKind.Iterable and \\\r\n\r\n~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _next_data(self)\r\n   1083             else:\r\n   1084                 del self._task_info[idx]\r\n-> 1085                 return self._process_data(data)\r\n   1086 \r\n   1087     def _try_put_index(self):\r\n\r\n~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_data(self, data)\r\n   1109         self._try_put_index()\r\n   1110         if isinstance(data, ExceptionWrapper):\r\n-> 1111             data.reraise()\r\n   1112         return data\r\n   1113 \r\n\r\n~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/_utils.py in reraise(self)\r\n    426             # have message field\r\n    427             raise self.exc_type(message=msg)\r\n--> 428         raise self.exc_type(msg)\r\n    429 \r\n    430 \r\n\r\nAssertionError: Caught AssertionError in DataLoader worker process 0.\r\nOriginal Traceback (most recent call last):\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop\r\n    data = fetcher.fetch(index)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch\r\n    data = [self.dataset[idx] for idx in possibly_batched_index]\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>\r\n    data = [self.dataset[idx] for idx in possibly_batched_index]\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1087, in __getitem__\r\n    format_kwargs=self._format_kwargs,\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1074, in _getitem\r\n    format_kwargs=format_kwargs,\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 890, in _convert_outputs\r\n    v = map_nested(command, v, **map_nested_kwargs)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 225, in map_nested\r\n    return function(data_struct)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 851, in command\r\n    return torch.tensor(x, **format_kwargs)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/warnings.py", line 101, in _showwarnmsg\r\n    _showwarnmsg_impl(msg)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/warnings.py", line 30, in _showwarnmsg_impl\r\n    file.write(text)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/lib/redirect.py", line 100, in new_write\r\n    cb(name, data)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/wandb_run.py", line 723, in _console_callback\r\n    self._backend.interface.publish_output(name, data)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 153, in publish_output\r\n    self._publish_output(o)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 158, in _publish_output\r\n    self._publish(rec)\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 456, in _publish\r\n    if self._process and not self._process.is_alive():\r\n  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/multiprocessing/process.py", line 134, in is_alive\r\n    assert self._parent_pid == os.getpid(), \'can only test a child process\'\r\nAssertionError: can only test a child process\r\n```\r\n\r\nAs a workaround I have commented line 456 and 457 in `/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py`'
 "Isn't it more the pytorch warning on the use of non-writable memory for tensor that trigger this here @lhoestq? (since it seems to be a warning triggered in `torch.tensor()`"
 'Yep this time this is a warning from pytorch that causes wandb to not work properly.\r\nCould this by a wandb issue ?'
 "Hi @timothyjlaurent @gaceladri \r\nIf you're running `transformers` from `master` you can try setting the env var `WAND_DISABLE=true` (from https://github.com/huggingface/transformers/pull/9896) and try again ?\r\nThis issue might be related to https://github.com/huggingface/transformers/issues/9623 "
 "I have commented the lines that cause my code break. I'm now seeing my reports on Wandb and my code does not break. I am training now, so I will check probably in 6 hours. I suppose that setting wandb disable will work as well."]
Using a dataset with a single 'text' field and a fast tokenizer in a jupyter notebook.

``` 
def tokenizer_fn(example):
    return tokenizer.batch_encode_plus(example['text'])

ds_tokenized = text_dataset.map(tokenizer_fn, batched=True, num_proc=6, remove_columns=['text'])
```


```
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 156, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1510, in _map_single
    for i in pbar:
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/tqdm/notebook.py", line 228, in __iter__
    for obj in super(tqdm_notebook, self).__iter__(*args, **kwargs):
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/tqdm/std.py", line 1186, in __iter__
    self.close()
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/tqdm/notebook.py", line 251, in close
    super(tqdm_notebook, self).close(*args, **kwargs)
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/tqdm/std.py", line 1291, in close
    fp_write('')
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/tqdm/std.py", line 1288, in fp_write
    self.fp.write(_unicode(s))
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/wandb/sdk/lib/redirect.py", line 91, in new_write
    cb(name, data)
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/wandb/sdk/wandb_run.py", line 598, in _console_callback
    self._backend.interface.publish_output(name, data)
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 146, in publish_output
    self._publish_output(o)
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 151, in _publish_output
    self._publish(rec)
  File "/home/jovyan/share/users/tlaurent/invitae-bert/ve/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 431, in _publish
    if self._process and not self._process.is_alive():
  File "/usr/lib/python3.6/multiprocessing/process.py", line 134, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
"""
```
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/847/reactions', 'total_count': 1, '+1': 1, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}