ERROR:root:mwparserfromhell
Because of the way Apache Beam works we indeed don't have access to the file name at this point in the code.
We'll have to use some tricks I think :p 

You can append `filepath` to `title` in `wikipedia.py:L512` for example. [[EDIT: it's L494 my bad]]
Then just do `try:...except:` on the  call of `_parse_and_clean_wikicode` L500 I guess.

Thanks for diving into this ! I tried it myself but I run out of memory on my laptop
As soon as we have the name of the file it should be easier to find what's wrong.
Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`


ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 Because of the way Apache Beam works we indeed don't have access to the file name at this point in the code.
We'll have to use some tricks I think :p 

You can append `filepath` to `title` in `wikipedia.py:L512` for example. [[EDIT: it's L494 my bad]]
Then just do `try:...except:` on the  call of `_parse_and_clean_wikicode` L500 I guess.

Thanks for diving into this ! I tried it myself but I run out of memory on my laptop
As soon as we have the name of the file it should be easier to find what's wrong.