Text dataset not working with large files
{'login': 'ksjae', 'id': 17930170, 'node_id': 'MDQ6VXNlcjE3OTMwMTcw', 'avatar_url': 'https://avatars.githubusercontent.com/u/17930170?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/ksjae', 'html_url': 'https://github.com/ksjae', 'followers_url': 'https://api.github.com/users/ksjae/followers', 'following_url': 'https://api.github.com/users/ksjae/following{/other_user}', 'gists_url': 'https://api.github.com/users/ksjae/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/ksjae/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/ksjae/subscriptions', 'organizations_url': 'https://api.github.com/users/ksjae/orgs', 'repos_url': 'https://api.github.com/users/ksjae/repos', 'events_url': 'https://api.github.com/users/ksjae/events{/privacy}', 'received_events_url': 'https://api.github.com/users/ksjae/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Seems like it works when setting ```block_size=2100000000``` or something arbitrarily large though.'
 'Can you give us some stats on the data files you use as inputs?'
 'Basically ~600MB txt files(UTF-8) * 59. \r\ncontents like ```안녕하세요, 이것은 예제로 한번 말해보는 텍스트입니다. 그냥 이렇다고요.<|endoftext|>\\n```\r\n\r\nAlso, it gets stuck for a loooong time at ```Testing the mapped function outputs```, for more than 12 hours(currently ongoing)'
 'It gets stuck while doing `.map()` ? Are you using multiprocessing ?\r\nIf you could provide a code snippet it could be very useful'
 'From transformers/examples/language-modeling/run-language-modeling.py :\r\n```\r\ndef get_dataset(\r\n    args: DataTrainingArguments,\r\n    tokenizer: PreTrainedTokenizer,\r\n    evaluate: bool = False,\r\n    cache_dir: Optional[str] = None,\r\n):\r\n    file_path = args.eval_data_file if evaluate else args.train_data_file\r\n    if True:\r\n        dataset = load_dataset("text", data_files=glob.glob(file_path), split=\'train\', use_threads=True, \r\n        ignore_verifications=True, save_infos=True, block_size=104857600)\r\n        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,\r\n                                                truncation=True, max_length=args.block_size), batched=True)\r\n        dataset.set_format(type=\'torch\', columns=[\'input_ids\'])\r\n        return dataset\r\n    if args.line_by_line:\r\n        return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)\r\n    else:\r\n        return TextDataset(\r\n            tokenizer=tokenizer,\r\n            file_path=file_path,\r\n            block_size=args.block_size,\r\n            overwrite_cache=args.overwrite_cache,\r\n            cache_dir=cache_dir,\r\n        )\r\n```\r\n\r\nNo, I\'m not using multiprocessing.'
 "I am not able to reproduce on my side :/\r\n\r\nCould you send the version of `datasets` and `pyarrow` you're using ?\r\nCould you try to update the lib and try again ?\r\nOr do you think you could try to reproduce it on google colab ?"
 "Huh, weird. It's fixed on my side too.\r\nBut now ```Caching processed dataset``` is taking forever - how can I disable it? Any flags?"
 "Right after `Caching processed dataset`, your function is applied to the dataset and there's a progress bar that shows how much time is left. How much time does it take for you ?\r\n\r\nAlso caching isn't supposed to slow down your processing. But if you still want to disable it you can do `.map(..., load_from_cache_file=False)`"
 'Ah, it’s much faster now(Takes around 15~20min). \r\nBTW, any way to set default tensor output as plain tensors with distributed training? The ragged tensors are incompatible with tpustrategy :('
 "> Ah, it’s much faster now(Takes around 15~20min).\r\n\r\nGlad to see that it's faster now. What did you change exactly ?\r\n\r\n> BTW, any way to set default tensor output as plain tensors with distributed training? The ragged tensors are incompatible with tpustrategy :(\r\n\r\nOh I didn't know about that. Feel free to open an issue to mention that.\r\nI guess what you can do for now is set the dataset format to numpy instead of tensorflow, and use a wrapper of the dataset that converts the numpy arrays to tf tensors.\r\n\r\n"
 ">>> Glad to see that it's faster now. What did you change exactly ?\r\nI don't know, it just worked...? Sorry I couldn't be more helpful.\r\n\r\nSetting with numpy array is a great idea! Thanks."]
```
Traceback (most recent call last):
  File "examples/language-modeling/run_language_modeling.py", line 333, in <module>
    main()
  File "examples/language-modeling/run_language_modeling.py", line 262, in main
    get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
  File "examples/language-modeling/run_language_modeling.py", line 144, in get_dataset
    dataset = load_dataset("text", data_files=file_path, split='train+test')
  File "/home/ksjae/.local/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/home/ksjae/.local/lib/python3.7/site-packages/datasets/builder.py", line 469, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/ksjae/.local/lib/python3.7/site-packages/datasets/builder.py", line 546, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/ksjae/.local/lib/python3.7/site-packages/datasets/builder.py", line 888, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
  File "/home/ksjae/.local/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/ksjae/.cache/huggingface/modules/datasets_modules/datasets/text/7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7/text.py", line 104, in _generate_tables
    convert_options=self.config.convert_options,
  File "pyarrow/_csv.pyx", line 714, in pyarrow._csv.read_csv
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
```

**pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)**

It gives the same message for both 200MB, 10GB .tx files but not for 700MB file.
Can't upload due to size & copyright problem. sorry.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/630/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}