Can datasets remove duplicated rows?
{'login': 'liuxinglan', 'id': 16516583, 'node_id': 'MDQ6VXNlcjE2NTE2NTgz', 'avatar_url': 'https://avatars.githubusercontent.com/u/16516583?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/liuxinglan', 'html_url': 'https://github.com/liuxinglan', 'followers_url': 'https://api.github.com/users/liuxinglan/followers', 'following_url': 'https://api.github.com/users/liuxinglan/following{/other_user}', 'gists_url': 'https://api.github.com/users/liuxinglan/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/liuxinglan/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/liuxinglan/subscriptions', 'organizations_url': 'https://api.github.com/users/liuxinglan/orgs', 'repos_url': 'https://api.github.com/users/liuxinglan/repos', 'events_url': 'https://api.github.com/users/liuxinglan/events{/privacy}', 'received_events_url': 'https://api.github.com/users/liuxinglan/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
['Hi ! For now this is probably the best option.\r\nWe might add a feature like this in the feature as well.\r\n\r\nDo you know any deduplication method that works on arbitrary big datasets without filling up RAM ?\r\nOtherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases'
 "Yes, I'd like to work on this feature once I'm done with #2500, but first I have to do some research, and see if the implementation wouldn't be too complex.\r\n\r\nIn the meantime, maybe [this lib](https://github.com/TomScheffers/pyarrow_ops) can help. However, note that this lib operates directly on pyarrow tables and relies only on `hash` to find duplicates (e.g. `-1` and `-2` have the same hash in Python 3, so this lib will treat them as duplicates), which doesn't make much sense."
 "> Hi ! For now this is probably the best option.\r\n> We might add a feature like this in the feature as well.\r\n> \r\n> Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?\r\n> Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases\r\n\r\nGreat if this is can be done. Thanks!!\r\n\r\nNot sure if you are asking me. In any case I don't know of any unfortunately :( in practice if data is really large we normally do it with spark (only for info. I understand this is not useful in developing this library..)"
 "Hello,\r\n\r\nI'm also interested in this feature.\r\nHas there been progress on this issue?\r\n\r\nCould we use a similar trick as above, but with a better hashing algorithm like SHA?\r\n\r\nWe could also use a [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter), should we care a lot about collision in this case?"
 'For reference, we can get a solution fairly easily if we assume that we can hold in memory all unique values. \r\n\r\n```python\r\nfrom datasets import Dataset\r\nfrom itertools import cycle\r\nfrom functools import partial\r\n\r\nmemory = set()\r\ndef is_unique(elem:Any , column: str, memory: set) -> bool:\r\n    if elem[column] in memory:\r\n        return False\r\n    else:\r\n        memory.add(elem[column])\r\n        return True\r\n\r\n# Example dataset\r\nds = Dataset.from_dict({"col1" : [sent for i, sent in zip(range(10), cycle(["apple", "orange", "pear"]))],\r\n                                      "col2": [i % 5 for i in range(10)]})\r\n\r\n# Drop duplicates in `ds` on "col1"\r\nds2 = ds.filter(partial(is_unique, column="col1", memory=memory))\r\n```\r\n\r\nOf course, we can improve the API so that we can introduce `Dataset.drop_duplicates`.\r\nFor the parallel version, we can use a shared memory set.'
 'An approach that works assuming you can hold the all the unique document hashes in memory:\r\n```python\r\nfrom datasets import load_dataset\r\n\r\ndef get_hash(example):\r\n    """Get hash of content field."""\r\n    return {"hash": hash(example["content"])} # can use any hashing function here\r\n    \r\ndef check_uniques(example, uniques):\r\n    """Check if current hash is still in set of unique hashes and remove if true."""\r\n    if example["hash"] in uniques:\r\n        uniques.remove(example["hash"])\r\n        return True\r\n    else:\r\n        return False\r\n\r\nds = load_dataset("some_dataset")\r\nds = ds.map(get_hash)\r\nuniques = set(ds.unique("hash"))\r\nds_filter = ds.filter(check_uniques, fn_kwargs={"uniques": uniques})\r\n```\r\nIf the `uniques` could be stored in arrow then no additional memory would used at all but I don\'t know if this is possible.\r\n']
**Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..


**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...


**Additional context**
no
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2514/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}