Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()
{'login': 'iluvvatar', 'id': 55381086, 'node_id': 'MDQ6VXNlcjU1MzgxMDg2', 'avatar_url': 'https://avatars.githubusercontent.com/u/55381086?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/iluvvatar', 'html_url': 'https://github.com/iluvvatar', 'followers_url': 'https://api.github.com/users/iluvvatar/followers', 'following_url': 'https://api.github.com/users/iluvvatar/following{/other_user}', 'gists_url': 'https://api.github.com/users/iluvvatar/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/iluvvatar/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/iluvvatar/subscriptions', 'organizations_url': 'https://api.github.com/users/iluvvatar/orgs', 'repos_url': 'https://api.github.com/users/iluvvatar/repos', 'events_url': 'https://api.github.com/users/iluvvatar/events{/privacy}', 'received_events_url': 'https://api.github.com/users/iluvvatar/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['To reproduce:\r\n\r\n```python\r\n>>> import datasets\r\n>>> datasets.get_dataset_split_names(\'MalakhovIlya/RuREBus\', config_name=\'raw_txt\')\r\nTraceback (most recent call last):\r\n  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 280, in get_dataset_config_info\r\n    for split_generator in builder._split_generators(\r\n  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 101, in _split_generators\r\n    decode_file_names(folder)\r\n  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 26, in decode_file_names\r\n    for root, dirs, files in os.walk(folder, topdown=False):\r\n  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/streaming.py", line 66, in wrapper\r\n    return function(*args, use_auth_token=use_auth_token, **kwargs)\r\nTypeError: xwalk() got an unexpected keyword argument \'topdown\'\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n  File "<stdin>", line 1, in <module>\r\n  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_split_names\r\n    info = get_dataset_config_info(\r\n  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 285, in get_dataset_config_info\r\n    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err\r\ndatasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.\r\n```\r\n\r\nIt\'s not related to the dataset viewer. Maybe @albertvillanova or @lhoestq could help more on this issue.'
 "Hi! This issue stems from the fact that `xwalk`, which is a streamable version of `os.walk`, doesn't support the `topdown` param due to `fsspec`'s `walk` also not supporting it, so fixing this issue could be tricky.  \r\n\r\n@MalakhovIlyaPavlovich You can avoid the error by tweaking your data processing and not using this param. (and `Path.rename`, which also cannot be streamed) "
 '@mariosasko thank you for your reply. I couldn\'t reproduce error showed by @severo either on Ubuntu 20.04.3 LTS, Windows 10 and Google Colab environments. But trying to avoid using os.walk(topdown=False) and Path.rename(), In _split_generators I replaced\r\n```\r\ndef decode_file_names(folder):\r\n    for root, dirs, files in os.walk(folder, topdown=False):\r\n        root = Path(root)\r\n        for file in files:\r\n            old_name = root / Path(file)\r\n            new_name = root / Path(\r\n                file.encode(\'cp437\').decode(\'cp866\'))\r\n            old_name.rename(new_name)\r\n        for dir in dirs:\r\n            old_name = root / Path(dir)\r\n            new_name = root / Path(dir.encode(\'cp437\').decode(\'cp866\'))\r\n            old_name.rename(new_name)\r\n\r\nfolder = dl_manager.download_and_extract(self._RAW_TXT_URLS)[\'raw_txt\']\r\ndecode_file_names(folder)\r\n```\r\nby\r\n```\r\ndef extract(zip_file_path):\r\n    p = Path(zip_file_path)\r\n    dest_dir = str(p.parent / \'extracted\' / p.stem)\r\n    os.makedirs(dest_dir, exist_ok=True)\r\n    with zipfile.ZipFile(zip_file_path) as archive:\r\n        for file_info in tqdm(archive.infolist(), desc=\'Extracting\'):\r\n            filename = file_info.filename.encode(\'cp437\').decode(\'cp866\')\r\n            target = os.path.join(dest_dir, *filename.split(\'/\'))\r\n            os.makedirs(os.path.dirname(target), exist_ok=True)\r\n            if not file_info.is_dir():\r\n                with archive.open(file_info) as source, open(target, \'wb\') as dest:\r\n                    shutil.copyfileobj(source, dest)\r\n    return dest_dir\r\n\r\nzip_file = dl_manager.download(self._RAW_TXT_URLS)[\'raw_txt\']\r\nif not is_url(zip_file):\r\n    folder = extract(zip_file)\r\nelse:\r\n    folder = None\r\n```\r\nand now everything works well except data viewer for "raw_txt" subset: dataset preview on hub shows "No data.". As far as I understand dl_manager.download returns original URL when we are calling datasets.get_dataset_split_names and my suspicions are that dataset viewer can do smth similar. I couldn\'t find information about how it works. I would be very grateful, if you could tell me how to fix this)'
 'This is what I get when I try to stream the `raw_txt` subset:\r\n```python\r\n>>> dset = load_dataset("MalakhovIlya/RuREBus", "raw_txt", split="raw_txt", streaming=True)\r\n>>> next(iter(dset))\r\nTraceback (most recent call last):\r\n  File "<stdin>", line 1, in <module>\r\nStopIteration\r\n```\r\nSo there is a bug in your script.'
 "streaming=True helped me to find solution. I fixed\r\n```\r\ndef extract(zip_file_path):\r\n    p = Path(zip_file_path)\r\n    dest_dir = str(p.parent / 'extracted' / p.stem)\r\n    os.makedirs(dest_dir, exist_ok=True)\r\n    with zipfile.ZipFile(zip_file_path) as archive:\r\n        for file_info in tqdm(archive.infolist(), desc='Extracting'):\r\n            filename = file_info.filename.encode('cp437').decode('cp866')\r\n            target = os.path.join(dest_dir, *filename.split('/'))\r\n            os.makedirs(os.path.dirname(target), exist_ok=True)\r\n            if not file_info.is_dir():\r\n                with archive.open(file_info) as source, open(target, 'wb') as dest:\r\n                    shutil.copyfileobj(source, dest)\r\n    return dest_dir\r\n\r\nzip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']\r\nfolder = extract(zip_file)\r\n```\r\nby \r\n```\r\nfolder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']\r\npath = os.path.join(folder, 'MED_txt/unparsed_txt')\r\nfor root, dirs, files in os.walk(path):\r\n    decoded_root_name = Path(root).name.encode('cp437').decode('cp866')\r\n```\r\n@mariosasko thank you for your help :)"]
## Dataset viewer issue for 'MalakhovIlya/RuREBus'

**Link:** https://huggingface.co/datasets/MalakhovIlya/RuREBus

**Description**
Using os.walk(topdown=False) in DatasetBuilder causes following error:
Status code:   400
Exception:     TypeError
Message:       xwalk() got an unexpected keyword argument 'topdown'
Couldn't find where "xwalk" come from. How can I fix this?

Am I the one who added this dataset ? Yes

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4138/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}