Support Zstandard compressed files
{'login': 'albertvillanova', 'id': 8515462, 'node_id': 'MDQ6VXNlcjg1MTU0NjI=', 'avatar_url': 'https://avatars.githubusercontent.com/u/8515462?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/albertvillanova', 'html_url': 'https://github.com/albertvillanova', 'followers_url': 'https://api.github.com/users/albertvillanova/followers', 'following_url': 'https://api.github.com/users/albertvillanova/following{/other_user}', 'gists_url': 'https://api.github.com/users/albertvillanova/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/albertvillanova/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/albertvillanova/subscriptions', 'organizations_url': 'https://api.github.com/users/albertvillanova/orgs', 'repos_url': 'https://api.github.com/users/albertvillanova/repos', 'events_url': 'https://api.github.com/users/albertvillanova/events{/privacy}', 'received_events_url': 'https://api.github.com/users/albertvillanova/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["> What if people want to run some tests without having zstandard ?\r\n> Usually what we do is add a decorator @require_zstandard for example\r\n\r\n@lhoestq I think I'm missing something here...\r\n\r\nTests are a *development* tool (to ensure we deliver a good quality lib), not something we offer to the end users of the lib. Users of the lib just `pip install datasets` and no tests are delivered with the lib (`tests` directory is outside the `src` code dir). \r\n\r\nOn the contrary, developers (contributors) of the lib do need to be able to run tests (TDD). And because of that, they are required to install datasets differently: `pip install -e .[dev]`, so that all required developing (and testing) dependencies are properly installed (included `zstandard`).\r\n\r\nApart from `zsatandard`, there are many other dev/test required dependencies for running tests, and we do not have a `@require_toto` for each and every of these dependencies in our tests: \r\n- `pytest` and `absl-py` (they are not dependencies in install_requires, but only in TEST_REQUIRE extras_require), \r\n- `boto3` (in test_filesystem.py), \r\n- `seqeval` (in test_metric_common.py), \r\n- `bs4` (used by eli5 and tested in test_hf_gcp.py)\r\n- ...\r\n\r\nSo IMHO, to run tests you should previously install datasets with dev or tests dependencies: either `pip install -e .[dev]` or `pip install -e .[tests]` (the latter to be used in CI testing-only part of the development cycle). And the tests should be written accordingly, assuming all tests dependencies are installed."
 "Hi !\r\nI was saying that because the other dependencies you mentioned are only required for _some_ tests. While here zstd is required for _all_ tests since it's imported in the conftest.py\r\nFeel free to keep it as it is right now, or maybe move the fixture to test_file_utils.py to allow users without zstd to run tests for their builders, dataset card etc. without issues"
 'Thank you ! I think we can merge now'
 '@lhoestq does this mean that the pile could have streaming support in the future? Afaik streaming doesnt support zstandard compressed type'
 '> @lhoestq does this mean that the pile could have streaming support in the future? Afaik streaming doesnt support zstandard compressed type\r\n\r\njust for reference, i tried to stream one of the `.zst` files from [the pile](https://the-eye.eu/public/AI/pile/) using\r\n\r\n```python\r\ndata_files = ["https://the-eye.eu/public/AI/pile/train/00.jsonl.zst"]\r\nstreamed_dataset = load_dataset(\'json\', split=\'train\', data_files=data_files, streaming=True)\r\n```\r\n\r\nand got the following error:\r\n\r\n```\r\nUsing custom data configuration default-4e71acadc389c254\r\n---------------------------------------------------------------------------\r\nNotImplementedError                       Traceback (most recent call last)\r\n/tmp/ipykernel_1187680/10848115.py in <module>\r\n      1 data_files = ["https://the-eye.eu/public/AI/pile/train/00.jsonl.zst"]\r\n      2 \r\n----> 3 streamed_dataset = load_dataset(\'json\', split=\'train\', data_files=data_files, streaming=True)\r\n      4 \r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)\r\n    835         # this extends the open and os.path.join functions for data streaming\r\n    836         extend_module_for_streaming(builder_instance.__module__, use_auth_token=use_auth_token)\r\n--> 837         return builder_instance.as_streaming_dataset(\r\n    838             split=split,\r\n    839             use_auth_token=use_auth_token,\r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/builder.py in as_streaming_dataset(self, split, base_path, use_auth_token)\r\n    922             data_dir=self.config.data_dir,\r\n    923         )\r\n--> 924         splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}\r\n    925         # By default, return all splits\r\n    926         if split is None:\r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/packaged_modules/json/json.py in _split_generators(self, dl_manager)\r\n     50         if not self.config.data_files:\r\n     51             raise ValueError(f"At least one data file must be specified, but got data_files={self.config.data_files}")\r\n---> 52         data_files = dl_manager.download_and_extract(self.config.data_files)\r\n     53         if isinstance(data_files, (str, list, tuple)):\r\n     54             files = data_files\r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in download_and_extract(self, url_or_urls)\r\n    140 \r\n    141     def download_and_extract(self, url_or_urls):\r\n--> 142         return self.extract(self.download(url_or_urls))\r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in extract(self, path_or_paths)\r\n    115 \r\n    116     def extract(self, path_or_paths):\r\n--> 117         urlpaths = map_nested(self._extract, path_or_paths, map_tuple=True)\r\n    118         return urlpaths\r\n    119 \r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types)\r\n    202         num_proc = 1\r\n    203     if num_proc <= 1 or len(iterable) <= num_proc:\r\n--> 204         mapped = [\r\n    205             _single_map_nested((function, obj, types, None, True))\r\n    206             for obj in utils.tqdm(iterable, disable=disable_tqdm)\r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)\r\n    203     if num_proc <= 1 or len(iterable) <= num_proc:\r\n    204         mapped = [\r\n--> 205             _single_map_nested((function, obj, types, None, True))\r\n    206             for obj in utils.tqdm(iterable, disable=disable_tqdm)\r\n    207         ]\r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)\r\n    141     # Singleton first to spare some computation\r\n    142     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):\r\n--> 143         return function(data_struct)\r\n    144 \r\n    145     # Reduce logging to keep things readable in multiprocessing with tqdm\r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _extract(self, urlpath)\r\n    119 \r\n    120     def _extract(self, urlpath):\r\n--> 121         protocol = self._get_extraction_protocol(urlpath)\r\n    122         if protocol is None:\r\n    123             # no extraction\r\n\r\n~/miniconda3/envs/hf/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _get_extraction_protocol(self, urlpath)\r\n    137         elif path.endswith(".zip"):\r\n    138             return "zip"\r\n--> 139         raise NotImplementedError(f"Extraction protocol for file at {urlpath} is not implemented yet")\r\n    140 \r\n    141     def download_and_extract(self, url_or_urls):\r\n\r\nNotImplementedError: Extraction protocol for file at https://the-eye.eu/public/AI/pile/train/00.jsonl.zst is not implemented yet\r\n```\r\n\r\ni\'m not sure whether @Shashi456 is referring to a fundamental limitation with "streaming" zstandard compression files or simply that we need to support the protocol in the streaming api of `datasets`\r\n\r\n'
 '@lewtun our streaming mode patches the Python `open` function. I could have a look tomorrow if it is easily implementable for this case.'
 "@lewtun, I have tested and yes, it is easily implementable. I've created a draft Pull Request with an implementation proposal: #2786."
 'thanks a lot @albertvillanova - now i can stream the pile :)']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/2578', 'html_url': 'https://github.com/huggingface/datasets/pull/2578', 'diff_url': 'https://github.com/huggingface/datasets/pull/2578.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/2578.patch', 'merged_at': datetime.datetime(2021, 7, 5, 10, 50, 27)}
Close #2572.

cc: @thomwolf 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2578/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}