ModuleNotFoundError: No module named 'apache_beam', when specific languages.
{'login': 'miyamonz', 'id': 6331508, 'node_id': 'MDQ6VXNlcjYzMzE1MDg=', 'avatar_url': 'https://avatars.githubusercontent.com/u/6331508?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/miyamonz', 'html_url': 'https://github.com/miyamonz', 'followers_url': 'https://api.github.com/users/miyamonz/followers', 'following_url': 'https://api.github.com/users/miyamonz/following{/other_user}', 'gists_url': 'https://api.github.com/users/miyamonz/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/miyamonz/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/miyamonz/subscriptions', 'organizations_url': 'https://api.github.com/users/miyamonz/orgs', 'repos_url': 'https://api.github.com/users/miyamonz/repos', 'events_url': 'https://api.github.com/users/miyamonz/events{/privacy}', 'received_events_url': 'https://api.github.com/users/miyamonz/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["Hi !\r\n\r\nApache Beam is a framework used to define data transformation pipelines. These pipeline can then be run in many runtimes: DataFlow, Spark, Flink, etc. There also exist a local runner called the DirectRunner.\r\nWikipedia is a dataset that requires some parsing, so to allow the processing to be run on this kind of runtime we're using Apache Beam.\r\n\r\nAt Hugging Face we've already processed certain versions of wikipedia (the `20200501.en` one for example) so that users can directly download the processed version instead of using Apache Beam to process it.\r\nHowever for the japanese language we haven't processed it so you'll have to run the processing on your side.\r\nSo you do need Apache Beam to process `20200501.ja`.\r\n\r\nYou can install Apache Beam with\r\n```\r\npip install apache-beam\r\n```\r\n\r\nI think we can probably improve the error message to let users know of this subtlety.\r\nWhat #498 implied is that Apache Beam is not needed when you process a dataset that doesn't use Apache Beam."
 'Thanks for your reply! \r\nI understood.\r\n\r\nI tried again with installing apache-beam, add ` beam_runner="DirectRunner"` and an anther `mwparserfromhell` is also required so I installed it.\r\nbut, it also failed. It exited 1 without error message.\r\n\r\n```py\r\nimport datasets\r\n# BTW, 20200501.ja doesn\'t exist at wikipedia, so I specified date argument\r\nwiki = datasets.load_dataset("wikipedia", language="ja", date="20210120", cache_dir="./datasets", beam_runner="DirectRunner")\r\nprint(wiki)\r\n```\r\nand its log is below\r\n```\r\nUsing custom data configuration 20210120.ja\r\nDownloading and preparing dataset wikipedia/20210120.ja-date=20210120,language=ja (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to ./datasets/wikipedia/20210120.ja-date=20210120,language=ja/0.0.0/4021357e28509391eab2f8300d9b689e7e8f3a877ebb3d354b01577d497ebc63...\r\nKilled\r\n```\r\n\r\nI also tried on another machine because it may caused by insufficient resources.\r\n```\r\n$ python main.py\r\nUsing custom data configuration 20210120.ja\r\nDownloading and preparing dataset wikipedia/20210120.ja-date=20210120,language=ja (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to ./datasets/wikipedia/20210120.ja-date=20210120,language=ja/0.0.0/4021357e28509391eab2f8300d9b689e7e8f3a877ebb3d354b01577d497ebc63...\r\n\r\nTraceback (most recent call last):\r\n  File "main.py", line 3, in <module>\r\n    wiki = datasets.load_dataset("wikipedia", language="ja", date="20210120", cache_dir="./datasets", beam_runner="DirectRunner")\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/datasets/load.py", line 609, in load_dataset\r\n    builder_instance.download_and_prepare(\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/datasets/builder.py", line 526, in download_and_prepare\r\n    self._download_and_prepare(\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/datasets/builder.py", line 1069, in _download_and_prepare\r\n    pipeline_results = pipeline.run()\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/pipeline.py", line 561, in run\r\n    return self.runner.run_pipeline(self, self._options)\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/direct/direct_runner.py", line 126, in run_pipeline\r\n    return runner.run_pipeline(pipeline, options)\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 182, in run_pipeline\r\n    self._latest_run_result = self.run_via_runner_api(\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 193, in run_via_runner_api\r\n    return self.run_stages(stage_context, stages)\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 358, in run_stages\r\n    stage_results = self._run_stage(\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 549, in _run_stage\r\n    last_result, deferred_inputs, fired_timers = self._run_bundle(\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 595, in _run_bundle\r\n    result, splits = bundle_manager.process_bundle(\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 888, in process_bundle\r\n    self._send_input_to_worker(process_bundle_id, transform_id, elements)\r\n  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 765, in _send_input_to_worker\r\n    data_out.write(byte_stream)\r\n  File "apache_beam/coders/stream.pyx", line 42, in apache_beam.coders.stream.OutputStream.write\r\n  File "apache_beam/coders/stream.pyx", line 47, in apache_beam.coders.stream.OutputStream.write\r\n  File "apache_beam/coders/stream.pyx", line 109, in apache_beam.coders.stream.OutputStream.extend\r\nAssertionError: OutputStream realloc failed.\r\n```\r\n\r\n'
 'Hi @miyamonz,\r\n\r\nI tried replicating this issue using the same snippet used by you. I am able to download the dataset without any issues, although I stopped it in the middle because the dataset is huge.\r\n\r\nBased on a similar issue [here](https://github.com/google-research/fixmatch/issues/23), it could be related to your environment setup, although I am just guessing here. Can you share these details?'
 'thanks for your reply and sorry for my late response.\r\n\r\n## environment\r\nmy local machine environment info\r\n- Ubuntu on WSL2\r\n\r\n`lsb_release -a`\r\n```\r\nNo LSB modules are available.\r\nDistributor ID: Ubuntu\r\nDescription:    Ubuntu 20.04.2 LTS\r\nRelease:        20.04\r\nCodename:       focal\r\n```\r\n\r\nRTX 2070 super\r\nInside WSL, there is no nvidia-msi command. I don\'t know why.\r\nBut, `torch.cuda.is_available()` is true and when I start something ML training code GPU usage is growing up, so I think it works.\r\n\r\nFrom PowerShell, there is nvidia-smi.exe and result is below.\r\n```\r\n+-----------------------------------------------------------------------------+\r\n| NVIDIA-SMI 470.05       Driver Version: 470.05       CUDA Version: 11.3     |\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|                               |                      |               MIG M. |\r\n|===============================+======================+======================|\r\n|   0  NVIDIA GeForce ... WDDM  | 00000000:09:00.0  On |                  N/A |\r\n|  0%   30C    P8    19W / 175W |    523MiB /  8192MiB |      3%      Default |\r\n|                               |                      |                  N/A |\r\n+-------------------------------+----------------------+----------------------+\r\n\r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                                  |\r\n|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\r\n|        ID   ID                                                   Usage      |\r\n|=============================================================================|\r\n|    0   N/A  N/A      1728    C+G   Insufficient Permissions        N/A      |\r\n|    0   N/A  N/A      3672    C+G   ...ekyb3d8bbwe\\YourPhone.exe    N/A      |\r\n|    0   N/A  N/A      6304    C+G   ...2txyewy\\TextInputHost.exe    N/A      |\r\n|    0   N/A  N/A      8648    C+G   C:\\Windows\\explorer.exe         N/A      |\r\n|    0   N/A  N/A      9536    C+G   ...y\\ShellExperienceHost.exe    N/A      |\r\n|    0   N/A  N/A     10668    C+G   ...5n1h2txyewy\\SearchApp.exe    N/A      |\r\n|    0   N/A  N/A     10948    C+G   ...artMenuExperienceHost.exe    N/A      |\r\n|    0   N/A  N/A     11988    C+G   ...8wekyb3d8bbwe\\Cortana.exe    N/A      |\r\n|    0   N/A  N/A     12464    C+G   ...cw5n1h2txyewy\\LockApp.exe    N/A      |\r\n|    0   N/A  N/A     13280    C+G   ...upport\\CEF\\Max Helper.exe    N/A      |\r\n|    0   N/A  N/A     15948    C+G   ...t\\GoogleIMEJaRenderer.exe    N/A      |\r\n|    0   N/A  N/A     16128    C+G   ...ram Files\\Slack\\Slack.exe    N/A      |\r\n|    0   N/A  N/A     19096    C+G   ...8bbwe\\WindowsTerminal.exe    N/A      |\r\n+-----------------------------------------------------------------------------+\r\n```\r\n\r\nI don\'t know what should I show in such a case. If it\'s not enough, please tell me some commands.\r\n\r\n---\r\n## what I did\r\nI surveyed more and I found 2 issues.\r\n\r\nAbout the first one, I wrote it as a new issue.\r\nhttps://github.com/huggingface/datasets/issues/2031\r\n\r\nThe error I mentioned in the previous comment above, which occurred on my local machine, is no longer occurring.\r\n\r\nBut, it still failed. In the previous comment, I wrote `AssertionError: OutputStream realloc failed.` happen on another machine. It also happens on my local machine.\r\n\r\nHere\'s what I\'ve tried.\r\n\r\nthe wikipedia.py downloads these xml.bz2 files based on dumpstatus.json\r\nIn Japanese Wikipedia dataset that I specified, it will download these 6 files.\r\n\r\n\r\n`https://dumps.wikimedia.org/jawiki/20210120/dumpstatus.json`\r\nand filtered json based on wikipedia.py is below.\r\n```json\r\n {\r\n   "jobs": {\r\n     "articlesmultistreamdump": {\r\n       "files": {\r\n         "jawiki-20210120-pages-articles-multistream1.xml-p1p114794.bz2": {\r\n           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream1.xml-p1p114794.bz2"\r\n         },\r\n         "jawiki-20210120-pages-articles-multistream2.xml-p114795p390428.bz2": {\r\n           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream2.xml-p114795p390428.bz2"\r\n         },\r\n         "jawiki-20210120-pages-articles-multistream3.xml-p390429p902407.bz2": {\r\n           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream3.xml-p390429p902407.bz2"\r\n         },\r\n         "jawiki-20210120-pages-articles-multistream4.xml-p902408p1721646.bz2": {\r\n           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream4.xml-p902408p1721646.bz2"\r\n         },\r\n         "jawiki-20210120-pages-articles-multistream5.xml-p1721647p2807947.bz2": {\r\n           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream5.xml-p1721647p2807947.bz2"\r\n         },\r\n         "jawiki-20210120-pages-articles-multistream6.xml-p2807948p4290013.bz2": {\r\n           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream6.xml-p2807948p4290013.bz2"\r\n         }\r\n       }\r\n     }\r\n   }\r\n }\r\n```\r\n\r\nSo, I tried running with fewer resources by modifying this line.\r\nhttps://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L524\r\nI changed it like this. just change filepaths list.\r\n`            | "Initialize" >> beam.Create(filepaths[:1])`\r\n\r\nand I added a print line inside for the loop of _extract_content.\r\nlike this `if(i % 100000 == 0): print(i)`\r\n\r\nfirst, without modification, it always stops after all _extract_content is done.\r\n\r\n- `filepaths[:1]` then it succeeded.\r\n- `filepaths[:2]` then it failed.\r\nI don\'t try all patterns because each pattern takes a long time.\r\n\r\n### my opinion\r\nIt seems it\'s successful when the entire file size is small.\r\n  \r\nso, at least it doesn\'t file-specific issue.\r\n\r\n\r\nI don\'t know it\'s true but I think when beam_writter writes into a file, it consumes memory depends on its entire file.\r\nbut It\'s correct Apache Beam\'s behavior? I\'m not familiar with this library.\r\n'
 "I don't know if this is related, but there is this issue on the wikipedia processing that you reported at #2031 (open PR is at #2037 ) .\r\nDoes the fix your proposed at #2037 helps in your case ?\r\n\r\nAnd for information, the DirectRunner of Apache Beam is not optimized for memory intensive tasks, so you must be right when you say that it uses the memory for the entire file."
 'the #2037 doesn\'t solve my problem directly, but I found the point!\r\n\r\nhttps://github.com/huggingface/datasets/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/datasets/wikipedia/wikipedia.py#L523\r\nthis `beam.transforms.Reshuffle()` cause the memory error.\r\n\r\nit makes sense if I consider the shuffle means. Beam\'s reshuffle seems need put all data in memory.\r\nPreviously I doubt that this line causes error, but at that time another bug showed in #2037 made error, so I can\'t found it.\r\n\r\nAnyway, I comment out this line, and run load_dataset, then it works!\r\n\r\n```python\r\nwiki = datasets.load_dataset(\r\n    "./wikipedia.py",\r\n    cache_dir="./datasets",\r\n    beam_runner="DirectRunner",\r\n    language="ja",\r\n    date="20210120",\r\n)["train"]\r\n```\r\n![image](https://user-images.githubusercontent.com/6331508/112283369-6a9f3300-8ccb-11eb-82e5-827bf7fddfb9.png)\r\n\r\nDataset has already shuffle function. https://github.com/huggingface/datasets/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/src/datasets/arrow_dataset.py#L2069\r\nSo, though I don\'t know it\'s difference correctly, but I think Beam\'s reshuffle isn\'t be needed. How do you think?'
 'The reshuffle is needed when you use parallelism.\r\nThe objective is to redistribute the articles evenly on the workers, since the `_extract_content` step generated many articles per file. By using reshuffle, we can split the processing of the articles of one file into several workers. Without reshuffle, all the articles of one file would be processed on the same worker that read the file, making the whole process take a very long time.'
 'Maybe the reshuffle step can be added only if the runner is not a DirectRunner ?']
```py
import datasets
wiki = datasets.load_dataset('wikipedia', '20200501.ja', cache_dir='./datasets')
```
then `ModuleNotFoundError: No module named 'apache_beam'` happend.

The error doesn't appear when it's '20200501.en'.
I don't know Apache Beam, but according to #498 it isn't necessary when it's saved to local. is it correct?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1790/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}