New Preprocessing Feature  - Deduplication [Request]
{'login': 'yuvalkirstain', 'id': 57996478, 'node_id': 'MDQ6VXNlcjU3OTk2NDc4', 'avatar_url': 'https://avatars.githubusercontent.com/u/57996478?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/yuvalkirstain', 'html_url': 'https://github.com/yuvalkirstain', 'followers_url': 'https://api.github.com/users/yuvalkirstain/followers', 'following_url': 'https://api.github.com/users/yuvalkirstain/following{/other_user}', 'gists_url': 'https://api.github.com/users/yuvalkirstain/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/yuvalkirstain/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/yuvalkirstain/subscriptions', 'organizations_url': 'https://api.github.com/users/yuvalkirstain/orgs', 'repos_url': 'https://api.github.com/users/yuvalkirstain/repos', 'events_url': 'https://api.github.com/users/yuvalkirstain/events{/privacy}', 'received_events_url': 'https://api.github.com/users/yuvalkirstain/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892865, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODY1', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/duplicate', 'name': 'duplicate', 'color': 'cfd3d7', 'default': True, 'description': 'This issue or pull request already exists'}
 {'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
["Hi! The [datasets_sql](https://github.com/mariosasko/datasets_sql) package lets you easily find distinct rows in a dataset (an example with `SELECT DISTINCT` is in the readme). Deduplication is (still) not part of the official API because it's hard to implement for datasets bigger than RAM while only using the native PyArrow ops.\r\n\r\n(Btw, this is a duplicate of https://github.com/huggingface/datasets/issues/2514)"]
**Is your feature request related to a problem? Please describe.**
Many large datasets are full of duplications and it has been shown that deduplicating datasets can lead to better performance while training, and more truthful evaluation at test-time.

A feature that allows one to easily deduplicate a dataset can be cool!

**Describe the solution you'd like**
We can define a function and keep only the first/last data-point that yields the value according to this function.

**Describe alternatives you've considered**
The clear alternative is to repeat a clear boilerplate every time someone want to deduplicate a dataset.

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4448/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}