Not enough disk space (Needed: Unknown size) when caching on a cluster
{'login': 'olinguyen', 'id': 4341867, 'node_id': 'MDQ6VXNlcjQzNDE4Njc=', 'avatar_url': 'https://avatars.githubusercontent.com/u/4341867?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/olinguyen', 'html_url': 'https://github.com/olinguyen', 'followers_url': 'https://api.github.com/users/olinguyen/followers', 'following_url': 'https://api.github.com/users/olinguyen/following{/other_user}', 'gists_url': 'https://api.github.com/users/olinguyen/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/olinguyen/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/olinguyen/subscriptions', 'organizations_url': 'https://api.github.com/users/olinguyen/orgs', 'repos_url': 'https://api.github.com/users/olinguyen/repos', 'events_url': 'https://api.github.com/users/olinguyen/events{/privacy}', 'received_events_url': 'https://api.github.com/users/olinguyen/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Hi ! 

What do you mean by "disk_usage(".").free` can't compute on the cluster's shared disk" exactly ?
Does it return 0 ? Yes, that's right. It shows 0 free space even though there is. I suspect it might have to do with permissions on the shared disk.

```python
>>> disk_usage(".")
usage(total=999999, used=999999, free=0)
``` That's an interesting behavior...
Do you know any other way to get the free space that works in your case ?
Also if it's a permission issue could you try fix the permissions and let mus know if that helped ? I think its an issue on the clusters end (unclear exactly why -- maybe something with docker containers?), will close the issue Were you able to figure it out? @philippnoah I had fixed it with a small hack where I patched `has_sufficient_disk_space` to always return `True`. you can do that with an import without having to modify the `datasets` package @olinguyen Thanks for the suggestion, it works but I had to to edit builder.py in the installed package. Can you please explain how were you able to do this using import? I was able to patch the builder code in my notebook before the load data call and it works. 
```
import datasets
datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
```
I'm running some experiments where I'm caching datasets on a cluster and accessing it through multiple compute nodes. However, I get an error when loading the cached dataset from the shared disk.

The exact error thrown:

```bash
>>> load_dataset(dataset, cache_dir="/path/to/cluster/shared/path")
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
```


[`utils.has_sufficient_disk_space`](https://github.com/huggingface/datasets/blob/8a03ab7d123a76ee744304f21ce868c75f411214/src/datasets/utils/py_utils.py#L332) fails on each job because of how the cluster system is designed (`disk_usage(".").free` can't compute on the cluster's shared disk).


This is exactly where the error gets thrown:
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L502

```python
if not utils.has_sufficient_disk_space(self.info.size_in_bytes or 0, directory=self._cache_dir_root):
    raise IOError(
          "Not enough disk space. Needed: {} (download: {}, generated: {}, post-processed: {})".format(
          utils.size_str(self.info.size_in_bytes or 0),
          utils.size_str(self.info.download_size or 0),
          utils.size_str(self.info.dataset_size or 0),
          utils.size_str(self.info.post_processing_size or 0),
       )
    )

```

What would be a good way to circumvent this? my current fix is to manually comment out that part, but that is not ideal. 
Would it be possible to pass a flag to skip this check on disk space?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1785/reactions', 'total_count': 3, '+1': 3, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}