Loading wikipedia 20200501.en throws pyarrow related error
> I just tried on my side and got no issues.
> When downloading the dataset again, did it crash at 10.7GB as well ?

Yes i have tried it multiple times on different machines. I am wondering if you could share the screenshot of your dependency versions and i will try to make them the same as yours?
**Problem description**
I am getting the following error when trying to load wikipedia/20200501.en dataset.

**Error log**
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, post-processed: Unknown size, total: 34.06 GiB) to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931...
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.6k/14.6k [00:00<00:00, 5.41MB/s]
Downloading:  59%|███████████████████████████████████████████████████████████████████████████████████████▊                                                              | 10.7G/18.3G [11:30<08:08, 15.5MB/s]
Dataset wikipedia downloaded and prepared to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931. Subsequent calls will reuse this data.
Traceback (most recent call last):
  File "load_wiki.py", line 2, in <module>
    ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 751, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 746, in as_dataset
    map_tuple=True,
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 142, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 763, in _build_single_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 835, in _as_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 215, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 236, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 171, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 302, in _get_dataset_from_filename
    pa_table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 324, in read_table
    pa_table = f.read_all()
  File "pyarrow/ipc.pxi", line 544, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Expected to be able to read 9176784 bytes for message body, got 4918712

**Detailed version info**
datasets==1.5.0
  - dataclasses [required: Any, installed: 0.8]
  - dill [required: Any, installed: 0.3.3]
  - fsspec [required: Any, installed: 0.8.7]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
  - huggingface-hub [required: <0.1.0, installed: 0.0.7]
    - filelock [required: Any, installed: 3.0.12]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
    - requests [required: Any, installed: 2.24.0]
      - certifi [required: >=2017.4.17, installed: 2020.6.20]
      - chardet [required: >=3.0.2,<4, installed: 3.0.4]
      - idna [required: >=2.5,<3, installed: 2.6]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
    - tqdm [required: Any, installed: 4.49.0]
  - importlib-metadata [required: Any, installed: 1.7.0]
    - zipp [required: >=0.5, installed: 3.1.0]
  - multiprocess [required: Any, installed: 0.70.11.1]
    - dill [required: >=0.3.3, installed: 0.3.3]
  - numpy [required: >=1.17, installed: 1.17.0]
  - pandas [required: Any, installed: 1.1.5]
    - numpy [required: >=1.15.4, installed: 1.17.0]
    - python-dateutil [required: >=2.7.3, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.2, installed: 2020.1]
  - pyarrow [required: >=0.17.1, installed: 3.0.0]
    - numpy [required: >=1.16.6, installed: 1.17.0]
  - requests [required: >=2.19.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.6]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - tqdm [required: >=4.27,<4.50.0, installed: 4.49.0]
  - xxhash [required: Any, installed: 2.0.0]

Loading wikipedia 20200501.en throws pyarrow related error 
 **Problem description**
I am getting the following error when trying to load wikipedia/20200501.en dataset.

**Error log**
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, post-processed: Unknown size, total: 34.06 GiB) to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931...
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.6k/14.6k [00:00<00:00, 5.41MB/s]
Downloading:  59%|███████████████████████████████████████████████████████████████████████████████████████▊                                                              | 10.7G/18.3G [11:30<08:08, 15.5MB/s]
Dataset wikipedia downloaded and prepared to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931. Subsequent calls will reuse this data.
Traceback (most recent call last):
  File "load_wiki.py", line 2, in <module>
    ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 751, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 746, in as_dataset
    map_tuple=True,
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 142, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 763, in _build_single_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 835, in _as_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 215, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 236, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 171, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 302, in _get_dataset_from_filename
    pa_table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 324, in read_table
    pa_table = f.read_all()
  File "pyarrow/ipc.pxi", line 544, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Expected to be able to read 9176784 bytes for message body, got 4918712

**Detailed version info**
datasets==1.5.0
  - dataclasses [required: Any, installed: 0.8]
  - dill [required: Any, installed: 0.3.3]
  - fsspec [required: Any, installed: 0.8.7]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
  - huggingface-hub [required: <0.1.0, installed: 0.0.7]
    - filelock [required: Any, installed: 3.0.12]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
    - requests [required: Any, installed: 2.24.0]
      - certifi [required: >=2017.4.17, installed: 2020.6.20]
      - chardet [required: >=3.0.2,<4, installed: 3.0.4]
      - idna [required: >=2.5,<3, installed: 2.6]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
    - tqdm [required: Any, installed: 4.49.0]
  - importlib-metadata [required: Any, installed: 1.7.0]
    - zipp [required: >=0.5, installed: 3.1.0]
  - multiprocess [required: Any, installed: 0.70.11.1]
    - dill [required: >=0.3.3, installed: 0.3.3]
  - numpy [required: >=1.17, installed: 1.17.0]
  - pandas [required: Any, installed: 1.1.5]
    - numpy [required: >=1.15.4, installed: 1.17.0]
    - python-dateutil [required: >=2.7.3, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.2, installed: 2020.1]
  - pyarrow [required: >=0.17.1, installed: 3.0.0]
    - numpy [required: >=1.16.6, installed: 1.17.0]
  - requests [required: >=2.19.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.6]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - tqdm [required: >=4.27,<4.50.0, installed: 4.49.0]
  - xxhash [required: Any, installed: 2.0.0]
 
 > I just tried on my side and got no issues.
> When downloading the dataset again, did it crash at 10.7GB as well ?

Yes i have tried it multiple times on different machines. I am wondering if you could share the screenshot of your dependency versions and i will try to make them the same as yours?