iter_archive on zipfiles with better compression type check
{'login': 'Mehdi2402', 'id': 56029953, 'node_id': 'MDQ6VXNlcjU2MDI5OTUz', 'avatar_url': 'https://avatars.githubusercontent.com/u/56029953?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/Mehdi2402', 'html_url': 'https://github.com/Mehdi2402', 'followers_url': 'https://api.github.com/users/Mehdi2402/followers', 'following_url': 'https://api.github.com/users/Mehdi2402/following{/other_user}', 'gists_url': 'https://api.github.com/users/Mehdi2402/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/Mehdi2402/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/Mehdi2402/subscriptions', 'organizations_url': 'https://api.github.com/users/Mehdi2402/orgs', 'repos_url': 'https://api.github.com/users/Mehdi2402/repos', 'events_url': 'https://api.github.com/users/Mehdi2402/events{/privacy}', 'received_events_url': 'https://api.github.com/users/Mehdi2402/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Hello @lhoestq, thank you for your answer.\r\n\r\nI don\'t use pytest a lot so I think I might need some help on it :) but I tried some tests for `streaming_download_manager.py` only. I don\'t know how to test `download_manager.py` since we need to use local files.\r\n\r\n# Comments : \r\n* In **download_manager.py** I removed some unnecessary imports after the simplification of `_get_extraction_protocol_local`.\r\n* In **streaming_download_manager** I moved the raised Error as suggested.\r\n \r\n### I also started some tests on `StreamingDownloadManager()` :\r\n* Used an existing zipfile url and added a new one that has a folder and many files : \r\n```python\r\nTEST_GG_DRIVE_ZIPPED_URL = "https://drive.google.com/uc?export=download&id=1k92sUfpHxKq8PXWRr7Y5aNHXwOCNUmqh"\r\nTEST_GG_DRIVE2_ZIPPED_URL = "https://drive.google.com/uc?export=download&id=1X4jyUBBbShyCRfD-vCO1ZvfqFXP3NEeU"\r\n``` \r\n* **For now is being tested :**\r\n     * Return type of the function  : should be tuple\r\n     * Files names\r\n     * Files content\r\n * Added an `xfail` test for the gzip file, because I get a `zipfile.BadZipFile exception`.\r\n\r\n\r\n * And lastly, changed the test for `_get_extraction_protocol_throws` since it was moved to `_extract` : \r\n ```diff\r\n@pytest.mark.xfail(raises=NotImplementedError)\r\ndef test_streaming_dl_manager_get_extraction_protocol_throws(urlpath):\r\n-    _get_extraction_protocol(urlpath)\r\n\r\n@pytest.mark.xfail(raises=NotImplementedError)\r\ndef test_streaming_dl_manager_get_extraction_protocol_throws(urlpath):\r\n+    StreamingDownloadManager()._extract(urlpath)\r\n```\r\n\r\n\r\n'
 'Hello,\r\nIn this Commit was taken into account all the comment escept the `test_download _manager.py`.\r\nI will work on that for the next commit.\r\n\r\nSorry again for being inactive lately in this PR.\r\n\r\n'
 'thanks a lot ! This CI seems to have import errors now though ?'
 "> thanks a lot ! This CI seems to have import errors now though ?\r\n\r\nYes sorry about that, it's due to a cyclic import I didn't pay attention to.\r\n\r\nWill fix that in the next Commit along with adding the tests to download_manager.\r\n\r\n"
 "في ثلاثاء، ٨ فبراير، ٢٠٢٢ في ٦:١٧ م، كتب EL MEHDI AGUNAOU <\n***@***.***>:\n\n> thanks a lot ! This CI seems to have import errors now though ?\n>\n> Yes sorry about that, it's due to a cyclic import I didn't pay attention\n> to.\n>\n> Will fix that in the next Commit along with adding the tests to\n> download_manager.\n>\n> —\n> Reply to this email directly, view it on GitHub\n> <https://github.com/huggingface/datasets/pull/3379#issuecomment-1032721249>,\n> or unsubscribe\n> <https://github.com/notifications/unsubscribe-auth/AR5LPDMJLZEKGVPKSD66VRLU2EXYDANCNFSM5JK6KTPA>\n> .\n> Triage notifications on the go with GitHub Mobile for iOS\n> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>\n> or Android\n> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.\n>\n> You are receiving this because you are subscribed to this thread.Message\n> ID: ***@***.***>\n>\n"]
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/3379', 'html_url': 'https://github.com/huggingface/datasets/pull/3379', 'diff_url': 'https://github.com/huggingface/datasets/pull/3379.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/3379.patch', 'merged_at': None}
Hello @lhoestq , thank you for your detailed answer on previous PR ! 
I made this new PR because I misused git on the previous one #3347.
Related issue #3272.

# Comments  : 

* For extension check I used the `_get_extraction_protocol` function in **download_manager.py** with a slight change and called it `_get_extraction_protocol_local`: 

**I removed this part :**
 ```python
    elif path.endswith(".tar.gz") or path.endswith(".tgz"):
        raise NotImplementedError(
            f"Extraction protocol for TAR archives like '{urlpath}' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead."
        )
```
**And also changed :** 
```diff
- extension = path.split(".")[-1]
+ extension = "tar" if path.endswith(".tar.gz") else path.split(".")[-1]
```
The reason for this is a compression like **.tar.gz** will be considered a **.gz** which is handled with **zipfile**, though **tar.gz** can only be opened using **tarfile**.

Please tell me if there's anything to change.

# Tasks : 
- [x] download_manager.py
- [x] streaming_download_manager.py 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3379/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}