Got pyarrow error when loading a dataset while adding special tokens into the tokenizer
{'login': 'yana-xuyan', 'id': 38536635, 'node_id': 'MDQ6VXNlcjM4NTM2NjM1', 'avatar_url': 'https://avatars.githubusercontent.com/u/38536635?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/yana-xuyan', 'html_url': 'https://github.com/yana-xuyan', 'followers_url': 'https://api.github.com/users/yana-xuyan/followers', 'following_url': 'https://api.github.com/users/yana-xuyan/following{/other_user}', 'gists_url': 'https://api.github.com/users/yana-xuyan/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/yana-xuyan/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/yana-xuyan/subscriptions', 'organizations_url': 'https://api.github.com/users/yana-xuyan/orgs', 'repos_url': 'https://api.github.com/users/yana-xuyan/repos', 'events_url': 'https://api.github.com/users/yana-xuyan/events{/privacy}', 'received_events_url': 'https://api.github.com/users/yana-xuyan/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892857, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODU3', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/bug', 'name': 'bug', 'color': 'd73a4a', 'default': True, 'description': "Something isn't working"}]
[]
['Hi,\r\n\r\nthe output of the tokenizers is treated specially in the lib to optimize the dataset size (see the code [here](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_writer.py#L138-L141)). It looks like that one of the values in a dictionary returned by the tokenizer is out of the assumed range.\r\nCan you please provide a minimal reproducible example for more help?'
 'Hi @yana-xuyan, thanks for reporting.\r\n\r\nAs clearly @mariosasko explained, `datasets` performs some optimizations in order to reduce the size of the dataset cache files. And one of them is storing the field `special_tokens_mask` as `int8`, which means that this field can only contain integers between `-128` to `127`. As your message error states, one of the values of this field is `50259`, and therefore it cannot be stored as an `int8`.\r\n\r\nMaybe we could implement a way to disable this optimization and allow using any integer value; although the size of the cache files would be much larger.'
 'I\'m facing same issue @mariosasko @albertvillanova \r\n\r\n```\r\nArrowInvalid: Integer value 50260 not in range: -128 to 127\r\n```\r\n\r\nTo reproduce:\r\n```python\r\nSPECIAL_TOKENS = [\'<bos>\',\'<eos>\',\'<speaker1>\',\'<speaker2>\',\'<pad>\']\r\nATTR_TO_SPECIAL_TOKEN = {\r\n    \'bos_token\': \'<bos>\', \r\n    \'eos_token\': \'<eos>\', \r\n    \'pad_token\': \'<pad>\',\r\n    \'additional_special_tokens\': [\'<speaker1>\', \'<speaker2>\']\r\n    }\r\n\r\ntokenizer = AutoTokenizer.from_pretrained("gpt2", use_fast=False)\r\nnum_added_tokens =tokenizer.add_special_tokens(ATTR_TO_SPECIAL_TOKEN)\r\nvocab_size = len(self.tokenizer.encoder) + num_added_tokens\r\nvocab =tokenizer.get_vocab()\r\n\r\npad_index = tokenizer.pad_token_id\r\neos_index = tokenizer.eos_token_id\r\nbos_index = tokenizer.bos_token_id\r\nspeaker1_index = vocab["<speaker1>"]\r\nspeaker2_index = vocab["<speaker2>"]\r\n```\r\n\r\n```python\r\ntokenizer.decode([\'50260\'])\r\n\'<speaker1>\'\r\n```'
 '@mariosasko \r\nI am hitting this bug in the Bert tokenizer too.  I see that @albertvillanova labeled this as a bug back in April.  Has there been a fix released yet?\r\nWhat I did for now is to just disable the optimization in the HF library.  @yana-xuyan and @thomas-happify, is that what you did and did that work for you?\r\n\r\n'
 'Hi @gregg-ADP, \r\n\r\nThis is still a bug.\r\n\r\nAs @albertvillanova has suggested, maybe it\'s indeed worth adding a variable to `config.py` to have a way to disable this behavior.\r\n\r\nIn the meantime, this forced optimization can be disabled by specifying `features` (of the returned examples) in the `map` call:\r\n```python\r\nfrom datasets import *\r\n... # dataset init\r\nds.map(process_example, features=Features({"special_tokens_mask": Sequence(Value("int32")), ... rest of the features}) \r\n```\r\n\r\ncc @lhoestq so he is also aware of this issue'
 'Thanks for the quick reply @mariosasko.  What I did was to changed the optimizer to use int32 instead of int8. \r\nWhat you\'re suggesting specifies the type for each feature explicitly without changing the HF code. This is definitely a better option.  However, we are hitting a new error later:\r\n```\r\n  File "/Users/ccccc/PycharmProjects/aaaa-ml/venv-source/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl\r\n    return forward_call(*input, **kwargs)\r\nTypeError: forward() got an unexpected keyword argument \'pos\'\r\n\r\n```\r\nWhere \'pos\' is the name of a new feature we added.  Do you agree that your way of fixing the optimizer issue will not fix our new issue?  If not, I will continue with this optimizer fix until we resolve our other issue.\r\n'
 "Hi @gwc4github,\r\n\r\nthe fix was merged a few minutes ago, and it doesn't require any changes on the user side (e.g. no need for specifying `features`). If you find time, feel free to install `datasets` from master with:\r\n```\r\npip install git+https://github.com/huggingface/datasets.git\r\n```\r\nand let us know if it works for your use case! "]
I added five more special tokens into the GPT2 tokenizer. But after that, when I try to pre-process the data using my previous code, I got an error shown below:

Traceback (most recent call last):
  File "/home/xuyan/anaconda3/envs/convqa/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1687, in _map_single
    writer.write(example)
  File "/home/xuyan/anaconda3/envs/convqa/lib/python3.7/site-packages/datasets/arrow_writer.py", line 296, in write
    self.write_on_file()
  File "/home/xuyan/anaconda3/envs/convqa/lib/python3.7/site-packages/datasets/arrow_writer.py", line 270, in write_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/xuyan/anaconda3/envs/convqa/lib/python3.7/site-packages/datasets/arrow_writer.py", line 108, in __arrow_array__
    out = out.cast(pa.list_(self.optimized_int_type))
  File "pyarrow/array.pxi", line 810, in pyarrow.lib.Array.cast
  File "/home/xuyan/anaconda3/envs/convqa/lib/python3.7/site-packages/pyarrow/compute.py", line 281, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 465, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 294, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 50259 not in range: -128 to 127

Do you have any idea about it?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2206/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}