Cached dataset overflowing disk space
{'login': 'BirgerMoell', 'id': 1704131, 'node_id': 'MDQ6VXNlcjE3MDQxMzE=', 'avatar_url': 'https://avatars.githubusercontent.com/u/1704131?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/BirgerMoell', 'html_url': 'https://github.com/BirgerMoell', 'followers_url': 'https://api.github.com/users/BirgerMoell/followers', 'following_url': 'https://api.github.com/users/BirgerMoell/following{/other_user}', 'gists_url': 'https://api.github.com/users/BirgerMoell/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/BirgerMoell/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/BirgerMoell/subscriptions', 'organizations_url': 'https://api.github.com/users/BirgerMoell/orgs', 'repos_url': 'https://api.github.com/users/BirgerMoell/repos', 'events_url': 'https://api.github.com/users/BirgerMoell/events{/privacy}', 'received_events_url': 'https://api.github.com/users/BirgerMoell/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["Hi! I'm transferring this issue over to `datasets`"
 "I'm using the datasets concatenate dataset to combine the datasets and then train.\r\ntrain_dataset = concatenate_datasets([dataset1, dataset2, common_voice_train])\r\n\r\n"
 "Hi @BirgerMoell.\r\n\r\nYou have several options:\r\n- to set caching to be stored on a different path location, other than the default one (`~/.cache/huggingface/datasets`):\r\n  - either setting the environment variable `HF_DATASETS_CACHE` with the path to the new cache location\r\n  - or by passing it with the parameter `cache_dir` when loading each of the datasets: `dataset = load_dataset(..., cache_dir=your_new_location)`\r\n\r\n  You can get all the information in the docs: https://huggingface.co/docs/datasets/loading_datasets.html#cache-directory\r\n- I wouldn't recommend disabling caching, because current implementation generates cache files anyway, although in a temporary directory and they are deleted when the session closes. See details here: https://huggingface.co/docs/datasets/processing.html#enable-or-disable-caching\r\n- You could alternatively load the datasets in streaming mode. This is a new feature which allows loading the datasets without downloading the entire files. More information here: https://huggingface.co/docs/datasets/dataset_streaming.html"
 'Hi @BirgerMoell,\r\n\r\nWe are planning to add a new feature to datasets, which could be interesting in your case: Add the option to delete temporary files (decompressed files) from the cache directory (see: #2481, #2604).\r\n\r\nWe will ping you once this feature is implemented, so that the size of your cache directory will be considerably reduced.']
I'm training a Swedish Wav2vec2 model on a Linux GPU and having issues that the huggingface cached dataset folder is completely filling up my disk space (I'm training on a dataset of around 500 gb).

The cache folder is 500gb (and now my disk space is full).

Is there a way to toggle caching or set the caching to be stored on a different device (I have another drive with 4 tb that could hold the caching files).

This might not technically be a bug, but I was unsure and I felt that the bug was the closest one.

Traceback (most recent call last):
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 186, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1983, in _map_single
    writer.finalize()
  File "/home/birger/miniconda3/envs/wav2vec2/lib/python3.7/site-packages/datasets/arrow_writer.py", line 418, in finalize
    self.pa_writer.close()
  File "pyarrow/ipc.pxi", line 402, in pyarrow.lib._CRecordBatchWriter.close
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
OSError: [Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on device
"""

The above exception was the direct cause of the following exception:

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2591/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}