Adding CC-100: Monolingual Datasets from Web Crawl Data
{'login': 'yjernite', 'id': 10469459, 'node_id': 'MDQ6VXNlcjEwNDY5NDU5', 'avatar_url': 'https://avatars.githubusercontent.com/u/10469459?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/yjernite', 'html_url': 'https://github.com/yjernite', 'followers_url': 'https://api.github.com/users/yjernite/followers', 'following_url': 'https://api.github.com/users/yjernite/following{/other_user}', 'gists_url': 'https://api.github.com/users/yjernite/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/yjernite/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/yjernite/subscriptions', 'organizations_url': 'https://api.github.com/users/yjernite/orgs', 'repos_url': 'https://api.github.com/users/yjernite/repos', 'events_url': 'https://api.github.com/users/yjernite/events{/privacy}', 'received_events_url': 'https://api.github.com/users/yjernite/received_events', 'type': 'User', 'site_admin': False}
[{'id': 2067376369, 'node_id': 'MDU6TGFiZWwyMDY3Mzc2MzY5', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/dataset%20request', 'name': 'dataset request', 'color': 'e99695', 'default': False, 'description': 'Requesting to add a new dataset'}]
{'login': 'abhishekkrthakur', 'id': 1183441, 'node_id': 'MDQ6VXNlcjExODM0NDE=', 'avatar_url': 'https://avatars.githubusercontent.com/u/1183441?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/abhishekkrthakur', 'html_url': 'https://github.com/abhishekkrthakur', 'followers_url': 'https://api.github.com/users/abhishekkrthakur/followers', 'following_url': 'https://api.github.com/users/abhishekkrthakur/following{/other_user}', 'gists_url': 'https://api.github.com/users/abhishekkrthakur/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/abhishekkrthakur/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/abhishekkrthakur/subscriptions', 'organizations_url': 'https://api.github.com/users/abhishekkrthakur/orgs', 'repos_url': 'https://api.github.com/users/abhishekkrthakur/repos', 'events_url': 'https://api.github.com/users/abhishekkrthakur/events{/privacy}', 'received_events_url': 'https://api.github.com/users/abhishekkrthakur/received_events', 'type': 'User', 'site_admin': False}
[{'login': 'abhishekkrthakur', 'id': 1183441, 'node_id': 'MDQ6VXNlcjExODM0NDE=', 'avatar_url': 'https://avatars.githubusercontent.com/u/1183441?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/abhishekkrthakur', 'html_url': 'https://github.com/abhishekkrthakur', 'followers_url': 'https://api.github.com/users/abhishekkrthakur/followers', 'following_url': 'https://api.github.com/users/abhishekkrthakur/following{/other_user}', 'gists_url': 'https://api.github.com/users/abhishekkrthakur/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/abhishekkrthakur/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/abhishekkrthakur/subscriptions', 'organizations_url': 'https://api.github.com/users/abhishekkrthakur/orgs', 'repos_url': 'https://api.github.com/users/abhishekkrthakur/repos', 'events_url': 'https://api.github.com/users/abhishekkrthakur/events{/privacy}', 'received_events_url': 'https://api.github.com/users/abhishekkrthakur/received_events', 'type': 'User', 'site_admin': False}]
cc @aconneau ;)  These dataset files are no longer available.  https://data.statmt.org/cc-100/ files provided in this link are no longer available. Can anybody fix that issue?
@abhishekkrthakur @yjernite  Hi ! Can you open an issue to report this problem ? This will help keep track of the fix :) Ok
## Adding a Dataset
- **Name:** CC-100: Monolingual Datasets from Web Crawl Data
- **Description:** https://twitter.com/alex_conneau/status/1321507120848625665
- **Paper:** https://arxiv.org/abs/1911.02116
- **Data:** http://data.statmt.org/cc-100/
- **Motivation:** A large scale multi-lingual language modeling dataset. Text is de-duplicated and filtered by how "Wikipedia-like" it is, hopefully helping avoid some of the worst parts of the common crawl.

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/773/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}