Downloading “reddit” dataset keeps timing out.
{'login': 'smeyerhot', 'id': 43877130, 'node_id': 'MDQ6VXNlcjQzODc3MTMw', 'avatar_url': 'https://avatars.githubusercontent.com/u/43877130?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/smeyerhot', 'html_url': 'https://github.com/smeyerhot', 'followers_url': 'https://api.github.com/users/smeyerhot/followers', 'following_url': 'https://api.github.com/users/smeyerhot/following{/other_user}', 'gists_url': 'https://api.github.com/users/smeyerhot/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/smeyerhot/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/smeyerhot/subscriptions', 'organizations_url': 'https://api.github.com/users/smeyerhot/orgs', 'repos_url': 'https://api.github.com/users/smeyerhot/repos', 'events_url': 'https://api.github.com/users/smeyerhot/events{/privacy}', 'received_events_url': 'https://api.github.com/users/smeyerhot/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892857, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODU3', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/bug', 'name': 'bug', 'color': 'd73a4a', 'default': True, 'description': "Something isn't working"}]
[]
['```\r\nUsing custom data configuration default\r\nDownloading and preparing dataset reddit/default (download: 2.93 GiB, generated: 17.64 GiB, post-processed: Unknown size, total: 20.57 GiB) to /Volumes/My Passport for Mac/og-chat-data/reddit/default/1.0.0/98ba5abea674d3178f7588aa6518a5510dc0c6fa8176d9653a3546d5afcb3969...\r\nDownloading: 13%\r\n403M/3.14G [44:39<2:27:09, 310kB/s]\r\n---------------------------------------------------------------------------\r\ntimeout                                   Traceback (most recent call last)\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in _error_catcher(self)\r\n    437             try:\r\n--> 438                 yield\r\n    439 \r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in read(self, amt, decode_content, cache_content)\r\n    518                 cache_content = False\r\n--> 519                 data = self._fp.read(amt) if not fp_closed else b""\r\n    520                 if (\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/http/client.py in read(self, amt)\r\n    458             b = bytearray(amt)\r\n--> 459             n = self.readinto(b)\r\n    460             return memoryview(b)[:n].tobytes()\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/http/client.py in readinto(self, b)\r\n    502         # (for example, reading in 1k chunks)\r\n--> 503         n = self.fp.readinto(b)\r\n    504         if not n and b:\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/socket.py in readinto(self, b)\r\n    703             try:\r\n--> 704                 return self._sock.recv_into(b)\r\n    705             except timeout:\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/ssl.py in recv_into(self, buffer, nbytes, flags)\r\n   1240                   self.__class__)\r\n-> 1241             return self.read(nbytes, buffer)\r\n   1242         else:\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/ssl.py in read(self, len, buffer)\r\n   1098             if buffer is not None:\r\n-> 1099                 return self._sslobj.read(len, buffer)\r\n   1100             else:\r\n\r\ntimeout: The read operation timed out\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nReadTimeoutError                          Traceback (most recent call last)\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/requests/models.py in generate()\r\n    757                 try:\r\n--> 758                     for chunk in self.raw.stream(chunk_size, decode_content=True):\r\n    759                         yield chunk\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in stream(self, amt, decode_content)\r\n    575             while not is_fp_closed(self._fp):\r\n--> 576                 data = self.read(amt=amt, decode_content=decode_content)\r\n    577 \r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in read(self, amt, decode_content, cache_content)\r\n    540                         # Content-Length are caught.\r\n--> 541                         raise IncompleteRead(self._fp_bytes_read, self.length_remaining)\r\n    542 \r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/contextlib.py in __exit__(self, type, value, traceback)\r\n    134             try:\r\n--> 135                 self.gen.throw(type, value, traceback)\r\n    136             except StopIteration as exc:\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in _error_catcher(self)\r\n    442                 # there is yet no clean way to get at it from this context.\r\n--> 443                 raise ReadTimeoutError(self._pool, None, "Read timed out.")\r\n    444 \r\n\r\nReadTimeoutError: HTTPSConnectionPool(host=\'zenodo.org\', port=443): Read timed out.\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nConnectionError                           Traceback (most recent call last)\r\n/var/folders/3f/md0t9sgj6rz8xy01fskttqdc0000gn/T/ipykernel_89016/1133441872.py in <module>\r\n      1 from datasets import load_dataset\r\n      2 \r\n----> 3 dataset = load_dataset("reddit", ignore_verifications=True, cache_dir="/Volumes/My Passport for Mac/og-chat-data")\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)\r\n    845 \r\n    846     # Download and prepare data\r\n--> 847     builder_instance.download_and_prepare(\r\n    848         download_config=download_config,\r\n    849         download_mode=download_mode,\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)\r\n    613                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")\r\n    614                     if not downloaded_from_gcs:\r\n--> 615                         self._download_and_prepare(\r\n    616                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs\r\n    617                         )\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)\r\n    669         split_dict = SplitDict(dataset_name=self.name)\r\n    670         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)\r\n--> 671         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)\r\n    672 \r\n    673         # Checksums verification\r\n\r\n~/.cache/huggingface/modules/datasets_modules/datasets/reddit/98ba5abea674d3178f7588aa6518a5510dc0c6fa8176d9653a3546d5afcb3969/reddit.py in _split_generators(self, dl_manager)\r\n     73     def _split_generators(self, dl_manager):\r\n     74         """Returns SplitGenerators."""\r\n---> 75         dl_path = dl_manager.download_and_extract(_URL)\r\n     76         return [\r\n     77             datasets.SplitGenerator(\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/download_manager.py in download_and_extract(self, url_or_urls)\r\n    287             extracted_path(s): `str`, extracted paths of given URL(s).\r\n    288         """\r\n--> 289         return self.extract(self.download(url_or_urls))\r\n    290 \r\n    291     def get_recorded_sizes_checksums(self):\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/download_manager.py in download(self, url_or_urls)\r\n    195 \r\n    196         start_time = datetime.now()\r\n--> 197         downloaded_path_or_paths = map_nested(\r\n    198             download_func,\r\n    199             url_or_urls,\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types)\r\n    194     # Singleton\r\n    195     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):\r\n--> 196         return function(data_struct)\r\n    197 \r\n    198     disable_tqdm = bool(logger.getEffectiveLevel() > logging.INFO) or not utils.is_progress_bar_enabled()\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/download_manager.py in _download(self, url_or_filename, download_config)\r\n    218             # append the relative path to the base_path\r\n    219             url_or_filename = url_or_path_join(self._base_path, url_or_filename)\r\n--> 220         return cached_path(url_or_filename, download_config=download_config)\r\n    221 \r\n    222     def iter_archive(self, path):\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)\r\n    286     if is_remote_url(url_or_filename):\r\n    287         # URL, so get it from the cache (downloading if necessary)\r\n--> 288         output_path = get_from_cache(\r\n    289             url_or_filename,\r\n    290             cache_dir=cache_dir,\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)\r\n    643                 ftp_get(url, temp_file)\r\n    644             else:\r\n--> 645                 http_get(\r\n    646                     url,\r\n    647                     temp_file,\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/file_utils.py in http_get(url, temp_file, proxies, resume_size, headers, cookies, timeout, max_retries)\r\n    451         disable=bool(logging.get_verbosity() == logging.NOTSET),\r\n    452     )\r\n--> 453     for chunk in response.iter_content(chunk_size=1024):\r\n    454         if chunk:  # filter out keep-alive new chunks\r\n    455             progress.update(len(chunk))\r\n\r\n/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/requests/models.py in generate()\r\n    763                     raise ContentDecodingError(e)\r\n    764                 except ReadTimeoutError as e:\r\n--> 765                     raise ConnectionError(e)\r\n    766             else:\r\n    767                 # Standard file-like object.\r\n\r\nConnectionError: HTTPSConnectionPool(host=\'zenodo.org\', port=443): Read timed out.\r\n```'
 'Hey @lhoestq should I try to fix this issue ?'
 'It also doesn\'t seem to be "smart caching" and I received an error about a file not being found...'
 'To be clear, the error I get when I try to "re-instantiate" the download after failure is: \r\n```\r\nOSError: Cannot find data file. \r\nOriginal error:\r\n[Errno 20] Not a directory: <HOME>/.cache/huggingface/datasets/downloads/1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c/corpus-webis-tldr-17.json\'\r\n```'
 "Here is a new error:\r\n```\r\nConnectionError: Couldn't reach https://zenodo.org/record/1043504/files/corpus-webis-tldr-17.zip?download=1\r\n```"
 "Hi ! Since https://github.com/huggingface/datasets/pull/2803 we've changed the time out from 10sec to 100sec.\r\nThis should prevent the `ReadTimeoutError`. Feel free to try it out by installing `datasets` from source\r\n```\r\npip install git+https://github.com/huggingface/datasets.git\r\n```\r\n\r\nWhen re-running your code you said you get a `OSError`, could you try deleting the file at the path returned by the error ? (the one after `[Errno 20] Not a directory:`). Ideally when a download fails you should be able to re-run it without error; there might be an issue here.\r\n\r\nFinally not sure what we can do about `ConnectionError`, this must be an issue from zenodo. If it happens you simply need to try again\r\n"
 "@lhoestq thanks for the update. The directory specified by the OSError ie. \r\n```\r\n1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c/corpus-webis-tldr-17.json \r\n```\r\n was not actually in that directory so I can't delete it. "
 'Oh, then could you try deleting the parent directory `1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c` instead ?\r\nThis way the download manager will know that it has to uncompress the data again'
 'It seems to have worked. It only took like 20min! I think the extra timeout length did the trick! One thing is that it downloaded a total of 41gb instead of 20gb but at least it finished. '
 'Great ! The timeout change will be available in the next release of `datasets` :)']
## Describe the bug
A clear and concise description of what the bug is.
Everytime I try and download the reddit dataset it times out before finishing and I have to try again.

There is some timeout error that I will post once it happens again.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("reddit", ignore_verifications=True, cache_dir="/Volumes/My Passport for Mac/og-chat-data")
```

## Expected results
A clear and concise description of the expected results.

I would expect the download to finish, or at least provide a parameter to extend the read timeout window.

## Actual results
Specify the actual results or traceback.

Shown below in error message.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:  1.11.0
- Platform: macOS 
- Python version: 3.9.6 (conda env)
- PyArrow version: N/A

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2820/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}