Keys should be unique error on code_search_net
{'login': 'thomwolf', 'id': 7353373, 'node_id': 'MDQ6VXNlcjczNTMzNzM=', 'avatar_url': 'https://avatars.githubusercontent.com/u/7353373?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/thomwolf', 'html_url': 'https://github.com/thomwolf', 'followers_url': 'https://api.github.com/users/thomwolf/followers', 'following_url': 'https://api.github.com/users/thomwolf/following{/other_user}', 'gists_url': 'https://api.github.com/users/thomwolf/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/thomwolf/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/thomwolf/subscriptions', 'organizations_url': 'https://api.github.com/users/thomwolf/orgs', 'repos_url': 'https://api.github.com/users/thomwolf/repos', 'events_url': 'https://api.github.com/users/thomwolf/events{/privacy}', 'received_events_url': 'https://api.github.com/users/thomwolf/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892857, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODU3', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/bug', 'name': 'bug', 'color': 'd73a4a', 'default': True, 'description': "Something isn't working"}]
[]
["Two questions:\r\n- with `datasets-cli env` we don't have any information on the dataset script version used. Should we give access to this somehow? Either as a note in the Error message or as an argument with the name of the dataset to `datasets-cli env`?\r\n- I don't really understand why the id is duplicated in the code of `code_search_net`, how can I debug this actually?"
 'Thanks for reporting. There was indeed an issue with the keys. The key was the addition of the file id and row id, which resulted in collisions. I just opened a PR to fix this at https://github.com/huggingface/datasets/pull/2555\r\n\r\nTo help users debug this kind of errors we could try to show a message like this\r\n```python\r\nDuplicateKeysError: both 42th and 1337th examples have the same keys `48`.\r\nPlease fix the dataset script at <path/to/the/dataset/script>\r\n```\r\n\r\nThis way users who what to look for if they want to debug this issue. I opened an issue to track this: https://github.com/huggingface/datasets/issues/2556'
 'and are we sure there are not a lot of datasets which are now broken with this change?'
 "Thanks to the dummy data, we know for sure that most of them work as expected.\r\n`code_search_net` wasn't caught because the dummy data only have one dummy data file while the dataset script can actually load several of them using `os.listdir`. Let me take a look at all the other datasets that use `os.listdir` to see if the keys are alright"
 'I found one issue on `fever` (PR here: https://github.com/huggingface/datasets/pull/2557)\r\nAll the other ones seem fine :)'
 "Hi! Got same error when loading other dataset:\r\n```python3\r\nload_dataset('wikicorpus', 'raw_en')\r\n```\r\n\r\ntb:\r\n```pytb\r\n---------------------------------------------------------------------------\r\nDuplicatedKeysError                       Traceback (most recent call last)\r\n/opt/conda/lib/python3.8/site-packages/datasets/builder.py in _prepare_split(self, split_generator)\r\n   1109                     example = self.info.features.encode_example(record)\r\n-> 1110                     writer.write(example, key)\r\n   1111             finally:\r\n\r\n/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py in write(self, example, key, writer_batch_size)\r\n    341             if self._check_duplicates:\r\n--> 342                 self.check_duplicate_keys()\r\n    343                 # Re-intializing to empty list for next batch\r\n\r\n/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py in check_duplicate_keys(self)\r\n    352             if hash in tmp_record:\r\n--> 353                 raise DuplicatedKeysError(key)\r\n    354             else:\r\n\r\nDuplicatedKeysError: FAILURE TO GENERATE DATASET !\r\nFound duplicate Key: 519\r\nKeys should be unique and deterministic in nature\r\n```\r\n\r\nVersion: datasets==1.11.0"
 'Fixed by #2555.'
 "The wikicorpus issue has been fixed by https://github.com/huggingface/datasets/pull/2844\r\n\r\nWe'll do a new release of `datasets` soon :)"]
## Describe the bug
Loading `code_search_net` seems not possible at the moment.

## Steps to reproduce the bug
```python
>>> load_dataset('code_search_net')
Downloading: 8.50kB [00:00, 3.09MB/s]                                                                                                                                           
Downloading: 19.1kB [00:00, 10.1MB/s]                                                                                                                                           
No config specified, defaulting to: code_search_net/all
Downloading and preparing dataset code_search_net/all (download: 4.77 GiB, generated: 5.99 GiB, post-processed: Unknown size, total: 10.76 GiB) to /Users/thomwolf/.cache/huggingface/datasets/code_search_net/all/1.0.0/b3e8278faf5d67da1d06981efbeac3b76a2900693bd2239bbca7a4a3b0d6e52a...
Traceback (most recent call last):         
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/builder.py", line 1067, in _prepare_split
    writer.write(example, key)
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/arrow_writer.py", line 343, in write
    self.check_duplicate_keys()
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/arrow_writer.py", line 354, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature
```

## Environment info
- `datasets` version: 1.8.1.dev0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 2.0.0

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2552/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}