[Arrow writer, Trivia_qa] Could not convert TagMe with type str: converting to null type
{'login': 'patrickvonplaten', 'id': 23423619, 'node_id': 'MDQ6VXNlcjIzNDIzNjE5', 'avatar_url': 'https://avatars.githubusercontent.com/u/23423619?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/patrickvonplaten', 'html_url': 'https://github.com/patrickvonplaten', 'followers_url': 'https://api.github.com/users/patrickvonplaten/followers', 'following_url': 'https://api.github.com/users/patrickvonplaten/following{/other_user}', 'gists_url': 'https://api.github.com/users/patrickvonplaten/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/patrickvonplaten/subscriptions', 'organizations_url': 'https://api.github.com/users/patrickvonplaten/orgs', 'repos_url': 'https://api.github.com/users/patrickvonplaten/repos', 'events_url': 'https://api.github.com/users/patrickvonplaten/events{/privacy}', 'received_events_url': 'https://api.github.com/users/patrickvonplaten/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
{'login': 'thomwolf', 'id': 7353373, 'node_id': 'MDQ6VXNlcjczNTMzNzM=', 'avatar_url': 'https://avatars.githubusercontent.com/u/7353373?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/thomwolf', 'html_url': 'https://github.com/thomwolf', 'followers_url': 'https://api.github.com/users/thomwolf/followers', 'following_url': 'https://api.github.com/users/thomwolf/following{/other_user}', 'gists_url': 'https://api.github.com/users/thomwolf/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/thomwolf/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/thomwolf/subscriptions', 'organizations_url': 'https://api.github.com/users/thomwolf/orgs', 'repos_url': 'https://api.github.com/users/thomwolf/repos', 'events_url': 'https://api.github.com/users/thomwolf/events{/privacy}', 'received_events_url': 'https://api.github.com/users/thomwolf/received_events', 'type': 'User', 'site_admin': False}
[{'login': 'thomwolf', 'id': 7353373, 'node_id': 'MDQ6VXNlcjczNTMzNzM=', 'avatar_url': 'https://avatars.githubusercontent.com/u/7353373?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/thomwolf', 'html_url': 'https://github.com/thomwolf', 'followers_url': 'https://api.github.com/users/thomwolf/followers', 'following_url': 'https://api.github.com/users/thomwolf/following{/other_user}', 'gists_url': 'https://api.github.com/users/thomwolf/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/thomwolf/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/thomwolf/subscriptions', 'organizations_url': 'https://api.github.com/users/thomwolf/orgs', 'repos_url': 'https://api.github.com/users/thomwolf/repos', 'events_url': 'https://api.github.com/users/thomwolf/events{/privacy}', 'received_events_url': 'https://api.github.com/users/thomwolf/received_events', 'type': 'User', 'site_admin': False}
 {'login': 'patrickvonplaten', 'id': 23423619, 'node_id': 'MDQ6VXNlcjIzNDIzNjE5', 'avatar_url': 'https://avatars.githubusercontent.com/u/23423619?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/patrickvonplaten', 'html_url': 'https://github.com/patrickvonplaten', 'followers_url': 'https://api.github.com/users/patrickvonplaten/followers', 'following_url': 'https://api.github.com/users/patrickvonplaten/following{/other_user}', 'gists_url': 'https://api.github.com/users/patrickvonplaten/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/patrickvonplaten/subscriptions', 'organizations_url': 'https://api.github.com/users/patrickvonplaten/orgs', 'repos_url': 'https://api.github.com/users/patrickvonplaten/repos', 'events_url': 'https://api.github.com/users/patrickvonplaten/events{/privacy}', 'received_events_url': 'https://api.github.com/users/patrickvonplaten/received_events', 'type': 'User', 'site_admin': False}
 {'login': 'lhoestq', 'id': 42851186, 'node_id': 'MDQ6VXNlcjQyODUxMTg2', 'avatar_url': 'https://avatars.githubusercontent.com/u/42851186?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lhoestq', 'html_url': 'https://github.com/lhoestq', 'followers_url': 'https://api.github.com/users/lhoestq/followers', 'following_url': 'https://api.github.com/users/lhoestq/following{/other_user}', 'gists_url': 'https://api.github.com/users/lhoestq/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lhoestq/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lhoestq/subscriptions', 'organizations_url': 'https://api.github.com/users/lhoestq/orgs', 'repos_url': 'https://api.github.com/users/lhoestq/repos', 'events_url': 'https://api.github.com/users/lhoestq/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lhoestq/received_events', 'type': 'User', 'site_admin': False}]
['Here the full error trace:\r\n\r\n```\r\nArrowInvalid                              Traceback (most recent call last)\r\n<ipython-input-1-7aaf3f011358> in <module>\r\n      1 import nlp\r\n      2 ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it\'s cached afterwards...\r\n----> 3 ds.map(lambda x: x, load_from_cache_file=False)\r\n\r\n~/python_bin/nlp/arrow_dataset.py in map(self, function, with_indices, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, arrow_schema, disable_nullable)\r\n    549\r\n    550         if update_data:\r\n--> 551             writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file\r\n    552\r\n    553             # Create new Dataset from buffer or file\r\n\r\n~/python_bin/nlp/arrow_writer.py in finalize(self, close_stream)\r\n    182     def finalize(self, close_stream=True):\r\n    183         if self.pa_writer is not None:\r\n--> 184             self.write_on_file()\r\n    185             self.pa_writer.close()\r\n    186         if close_stream:\r\n\r\n~/python_bin/nlp/arrow_writer.py in write_on_file(self)\r\n    104         """\r\n    105         if self.current_rows:\r\n--> 106             pa_array = pa.array(self.current_rows, type=self._type)\r\n    107             first_example = pa.array(self.current_rows[0:1], type=self._type)[0]\r\n    108             # Sanity check\r\n\r\n~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()\r\n\r\n~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()\r\n\r\n~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()\r\n\r\nArrowInvalid: Could not convert TagMe with type str: converting to null type\r\n```'
 'Actually thinking a bit more about it, it\'s probably a data sample that is not correct in `trivia_qa`. But I\'m a bit surprised though that we managed to write it in .arrow format and now cannot write it anymore after an "identity" mapping.'
 "I don't have this error :x"
 'Interesting, maybe I have a very old cache of trivia_qa...thanks for checking'
 "I'm running it right now on colab to double check"
 'Actually, I know what the problem is...I\'m quite sure it\'s a bug. Here we take some test inputs: https://github.com/huggingface/nlp/blob/0e0ef12c14d2175e0b0bd7d8aa814b09e2cd7e1f/src/nlp/arrow_dataset.py#L472\r\n\r\nIt might be that in the test inputs, a `Sequence` type value is an emtpy list. So in my case I have `ds[0]["entity_pages\']["wiki_context"] = []`. => this leads to an `arrow_schema` equal to `null`  for `["entity_pages\']["wiki_context"]` => see line: https://github.com/huggingface/nlp/blob/0e0ef12c14d2175e0b0bd7d8aa814b09e2cd7e1f/src/nlp/arrow_dataset.py#L501  instead of list of string which it should for other examples. \r\n\r\nGuess it\'s an edge case, but it can happen.'
 "Good point, I think the schema should be infered at the writing stage where we have a `writer_batch_size` number of examples (typically 10k) so it's even less likely to run into this scenario."]
Running the following code 

```
import nlp
ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
ds.map(lambda x: x, load_from_cache_file=False)
```

triggers a `ArrowInvalid: Could not convert TagMe with type str: converting to null type` error.

On the other hand if we remove a certain column of `trivia_qa` which seems responsible for the bug, it works:

```
import nlp
ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
ds.map(lambda x: x, remove_columns=["entity_pages"], load_from_cache_file=False)
```

. Seems quite hard to debug what's going on here... @lhoestq @thomwolf - do you have a good first guess what the problem could be?

**Note** BTW: I think this could be a good test to check that the datasets work correctly: Take a tiny portion of the dataset and check that it can be written correctly.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/211/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}