ModuleNotFoundError: No module named 'apache_beam', when specific languages.
thanks for your reply and sorry for my late response.

## environment
my local machine environment info
- Ubuntu on WSL2

`lsb_release -a`
```
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal
```

RTX 2070 super
Inside WSL, there is no nvidia-msi command. I don't know why.
But, `torch.cuda.is_available()` is true and when I start something ML training code GPU usage is growing up, so I think it works.

From PowerShell, there is nvidia-smi.exe and result is below.
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.05       Driver Version: 470.05       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:09:00.0  On |                  N/A |
|  0%   30C    P8    19W / 175W |    523MiB /  8192MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1728    C+G   Insufficient Permissions        N/A      |
|    0   N/A  N/A      3672    C+G   ...ekyb3d8bbwe\YourPhone.exe    N/A      |
|    0   N/A  N/A      6304    C+G   ...2txyewy\TextInputHost.exe    N/A      |
|    0   N/A  N/A      8648    C+G   C:\Windows\explorer.exe         N/A      |
|    0   N/A  N/A      9536    C+G   ...y\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     10668    C+G   ...5n1h2txyewy\SearchApp.exe    N/A      |
|    0   N/A  N/A     10948    C+G   ...artMenuExperienceHost.exe    N/A      |
|    0   N/A  N/A     11988    C+G   ...8wekyb3d8bbwe\Cortana.exe    N/A      |
|    0   N/A  N/A     12464    C+G   ...cw5n1h2txyewy\LockApp.exe    N/A      |
|    0   N/A  N/A     13280    C+G   ...upport\CEF\Max Helper.exe    N/A      |
|    0   N/A  N/A     15948    C+G   ...t\GoogleIMEJaRenderer.exe    N/A      |
|    0   N/A  N/A     16128    C+G   ...ram Files\Slack\Slack.exe    N/A      |
|    0   N/A  N/A     19096    C+G   ...8bbwe\WindowsTerminal.exe    N/A      |
+-----------------------------------------------------------------------------+
```

I don't know what should I show in such a case. If it's not enough, please tell me some commands.

---
## what I did
I surveyed more and I found 2 issues.

About the first one, I wrote it as a new issue.
https://github.com/huggingface/datasets/issues/2031

The error I mentioned in the previous comment above, which occurred on my local machine, is no longer occurring.

But, it still failed. In the previous comment, I wrote `AssertionError: OutputStream realloc failed.` happen on another machine. It also happens on my local machine.

Here's what I've tried.

the wikipedia.py downloads these xml.bz2 files based on dumpstatus.json
In Japanese Wikipedia dataset that I specified, it will download these 6 files.


`https://dumps.wikimedia.org/jawiki/20210120/dumpstatus.json`
and filtered json based on wikipedia.py is below.
```json
 {
   "jobs": {
     "articlesmultistreamdump": {
       "files": {
         "jawiki-20210120-pages-articles-multistream1.xml-p1p114794.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream1.xml-p1p114794.bz2"
         },
         "jawiki-20210120-pages-articles-multistream2.xml-p114795p390428.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream2.xml-p114795p390428.bz2"
         },
         "jawiki-20210120-pages-articles-multistream3.xml-p390429p902407.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream3.xml-p390429p902407.bz2"
         },
         "jawiki-20210120-pages-articles-multistream4.xml-p902408p1721646.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream4.xml-p902408p1721646.bz2"
         },
         "jawiki-20210120-pages-articles-multistream5.xml-p1721647p2807947.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream5.xml-p1721647p2807947.bz2"
         },
         "jawiki-20210120-pages-articles-multistream6.xml-p2807948p4290013.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream6.xml-p2807948p4290013.bz2"
         }
       }
     }
   }
 }
```

So, I tried running with fewer resources by modifying this line.
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L524
I changed it like this. just change filepaths list.
`            | "Initialize" >> beam.Create(filepaths[:1])`

and I added a print line inside for the loop of _extract_content.
like this `if(i % 100000 == 0): print(i)`

first, without modification, it always stops after all _extract_content is done.

- `filepaths[:1]` then it succeeded.
- `filepaths[:2]` then it failed.
I don't try all patterns because each pattern takes a long time.

### my opinion
It seems it's successful when the entire file size is small.
  
so, at least it doesn't file-specific issue.


I don't know it's true but I think when beam_writter writes into a file, it consumes memory depends on its entire file.
but It's correct Apache Beam's behavior? I'm not familiar with this library.

```py
import datasets
wiki = datasets.load_dataset('wikipedia', '20200501.ja', cache_dir='./datasets')
```
then `ModuleNotFoundError: No module named 'apache_beam'` happend.

The error doesn't appear when it's '20200501.en'.
I don't know Apache Beam, but according to #498 it isn't necessary when it's saved to local. is it correct?
ModuleNotFoundError: No module named 'apache_beam', when specific languages. 
 ```py
import datasets
wiki = datasets.load_dataset('wikipedia', '20200501.ja', cache_dir='./datasets')
```
then `ModuleNotFoundError: No module named 'apache_beam'` happend.

The error doesn't appear when it's '20200501.en'.
I don't know Apache Beam, but according to #498 it isn't necessary when it's saved to local. is it correct? 
 thanks for your reply and sorry for my late response.

## environment
my local machine environment info
- Ubuntu on WSL2

`lsb_release -a`
```
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal
```

RTX 2070 super
Inside WSL, there is no nvidia-msi command. I don't know why.
But, `torch.cuda.is_available()` is true and when I start something ML training code GPU usage is growing up, so I think it works.

From PowerShell, there is nvidia-smi.exe and result is below.
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.05       Driver Version: 470.05       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:09:00.0  On |                  N/A |
|  0%   30C    P8    19W / 175W |    523MiB /  8192MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1728    C+G   Insufficient Permissions        N/A      |
|    0   N/A  N/A      3672    C+G   ...ekyb3d8bbwe\YourPhone.exe    N/A      |
|    0   N/A  N/A      6304    C+G   ...2txyewy\TextInputHost.exe    N/A      |
|    0   N/A  N/A      8648    C+G   C:\Windows\explorer.exe         N/A      |
|    0   N/A  N/A      9536    C+G   ...y\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     10668    C+G   ...5n1h2txyewy\SearchApp.exe    N/A      |
|    0   N/A  N/A     10948    C+G   ...artMenuExperienceHost.exe    N/A      |
|    0   N/A  N/A     11988    C+G   ...8wekyb3d8bbwe\Cortana.exe    N/A      |
|    0   N/A  N/A     12464    C+G   ...cw5n1h2txyewy\LockApp.exe    N/A      |
|    0   N/A  N/A     13280    C+G   ...upport\CEF\Max Helper.exe    N/A      |
|    0   N/A  N/A     15948    C+G   ...t\GoogleIMEJaRenderer.exe    N/A      |
|    0   N/A  N/A     16128    C+G   ...ram Files\Slack\Slack.exe    N/A      |
|    0   N/A  N/A     19096    C+G   ...8bbwe\WindowsTerminal.exe    N/A      |
+-----------------------------------------------------------------------------+
```

I don't know what should I show in such a case. If it's not enough, please tell me some commands.

---
## what I did
I surveyed more and I found 2 issues.

About the first one, I wrote it as a new issue.
https://github.com/huggingface/datasets/issues/2031

The error I mentioned in the previous comment above, which occurred on my local machine, is no longer occurring.

But, it still failed. In the previous comment, I wrote `AssertionError: OutputStream realloc failed.` happen on another machine. It also happens on my local machine.

Here's what I've tried.

the wikipedia.py downloads these xml.bz2 files based on dumpstatus.json
In Japanese Wikipedia dataset that I specified, it will download these 6 files.


`https://dumps.wikimedia.org/jawiki/20210120/dumpstatus.json`
and filtered json based on wikipedia.py is below.
```json
 {
   "jobs": {
     "articlesmultistreamdump": {
       "files": {
         "jawiki-20210120-pages-articles-multistream1.xml-p1p114794.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream1.xml-p1p114794.bz2"
         },
         "jawiki-20210120-pages-articles-multistream2.xml-p114795p390428.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream2.xml-p114795p390428.bz2"
         },
         "jawiki-20210120-pages-articles-multistream3.xml-p390429p902407.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream3.xml-p390429p902407.bz2"
         },
         "jawiki-20210120-pages-articles-multistream4.xml-p902408p1721646.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream4.xml-p902408p1721646.bz2"
         },
         "jawiki-20210120-pages-articles-multistream5.xml-p1721647p2807947.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream5.xml-p1721647p2807947.bz2"
         },
         "jawiki-20210120-pages-articles-multistream6.xml-p2807948p4290013.bz2": {
           "url": "/jawiki/20210120/jawiki-20210120-pages-articles-multistream6.xml-p2807948p4290013.bz2"
         }
       }
     }
   }
 }
```

So, I tried running with fewer resources by modifying this line.
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L524
I changed it like this. just change filepaths list.
`            | "Initialize" >> beam.Create(filepaths[:1])`

and I added a print line inside for the loop of _extract_content.
like this `if(i % 100000 == 0): print(i)`

first, without modification, it always stops after all _extract_content is done.

- `filepaths[:1]` then it succeeded.
- `filepaths[:2]` then it failed.
I don't try all patterns because each pattern takes a long time.

### my opinion
It seems it's successful when the entire file size is small.
  
so, at least it doesn't file-specific issue.


I don't know it's true but I think when beam_writter writes into a file, it consumes memory depends on its entire file.
but It's correct Apache Beam's behavior? I'm not familiar with this library.
