wikipedia dataset incomplete
{'login': 'ChrisDelClea', 'id': 19912393, 'node_id': 'MDQ6VXNlcjE5OTEyMzkz', 'avatar_url': 'https://avatars.githubusercontent.com/u/19912393?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/ChrisDelClea', 'html_url': 'https://github.com/ChrisDelClea', 'followers_url': 'https://api.github.com/users/ChrisDelClea/followers', 'following_url': 'https://api.github.com/users/ChrisDelClea/following{/other_user}', 'gists_url': 'https://api.github.com/users/ChrisDelClea/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/ChrisDelClea/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/ChrisDelClea/subscriptions', 'organizations_url': 'https://api.github.com/users/ChrisDelClea/orgs', 'repos_url': 'https://api.github.com/users/ChrisDelClea/repos', 'events_url': 'https://api.github.com/users/ChrisDelClea/events{/privacy}', 'received_events_url': 'https://api.github.com/users/ChrisDelClea/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Hi !
From what pickle file fo you get this ?
I guess you mean the dataset loaded using `load_dataset` ? yes sorry, I used the `load_dataset`function and saved the data to a pickle file so I don't always have to reload it and are able to work offline.  The wikipedia articles are processed using the `mwparserfromhell` library. Even if it works well in most cases, such issues can happen unfortunately. You can find the repo here: https://github.com/earwig/mwparserfromhell

There also exist other datasets based on wikipedia that were processed differently (and are often cleaner) such as `wiki40b`.

 ok great. Thank you, @lhoestq. 
Hey guys,

I am using the https://github.com/huggingface/datasets/tree/master/datasets/wikipedia dataset.
Unfortunately, I found out that there is an incompleteness for the German dataset.
 For reasons unknown to me, the number of inhabitants has been removed from many pages:
Thorey-sur-Ouche has 128 inhabitants according to the webpage (https://de.wikipedia.org/wiki/Thorey-sur-Ouche).
The pickle file however shows: französische Gemeinde mit  Einwohnern (Stand).
 Is it possible to fix this?

Best regards 
Chris

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1759/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}