Filter on dataset too much slowww
{'login': 'ayubSubhaniya', 'id': 20911334, 'node_id': 'MDQ6VXNlcjIwOTExMzM0', 'avatar_url': 'https://avatars.githubusercontent.com/u/20911334?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/ayubSubhaniya', 'html_url': 'https://github.com/ayubSubhaniya', 'followers_url': 'https://api.github.com/users/ayubSubhaniya/followers', 'following_url': 'https://api.github.com/users/ayubSubhaniya/following{/other_user}', 'gists_url': 'https://api.github.com/users/ayubSubhaniya/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/ayubSubhaniya/subscriptions', 'organizations_url': 'https://api.github.com/users/ayubSubhaniya/orgs', 'repos_url': 'https://api.github.com/users/ayubSubhaniya/repos', 'events_url': 'https://api.github.com/users/ayubSubhaniya/events{/privacy}', 'received_events_url': 'https://api.github.com/users/ayubSubhaniya/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["When I use the filter on the arrow table directly, it works like butter. But I can't find a way to update the table in `Dataset` object.\r\n\r\n```\r\nds_table = dataset.data.filter(mask=dataset['flag'])\r\n```"
 '@thomwolf @lhoestq can you guys please take a look and recommend some solution.'
 "Hi ! Currently the filter method reads the dataset batch by batch to write a new, filtered, arrow file on disk. Therefore all the reading + writing can take some time.\r\nUsing a mask directly on the arrow table doesn't do any read or write operation therefore it's way quicker.\r\n\r\nReplacing the old table by the new one should do the job:\r\n```python\r\ndataset._data = dataset._data.filter(...)\r\n```\r\n\r\nNote: this is a **workaround** and in general users shouldn't have to do that. In particular if you did some `shuffle` or `select` before that then it would not work correctly since the indices mapping (index from `__getitem__` -> index in the table) would not be valid anymore. But if you haven't done any `shuffle`, `select`, `shard`, `train_test_split` etc. then it should work.\r\n\r\nIdeally it would be awesome to update the filter function to allow masking this way !\r\nIf you would like to give it a shot I will be happy to help :) "
 'Yes, would be happy to contribute. Thanks'
 "Hi @lhoestq @ayubSubhaniya,\r\n\r\nIf there's no progress on this one, can I try working on it?\r\n\r\nThanks,\r\nGunjan"
 'Sure @gchhablani feel free to start working on it, this would be very appreciated :)\r\nThis feature is would be really awesome, especially since arrow allows to mask really quickly and without having to rewrite the dataset on disk']
I have a dataset with 50M rows.
For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 12mins. I used `map()` with batch size 1024 and multi-process with 96 processes.

When I applied the `filter()` function it is taking too much time. I need to filter sequences based on a boolean column.
Below are the variants I tried.
1. filter() with batch size 1024, single process (takes roughly 3 hr)
2. filter() with batch size 1024, 96 processes (takes 5-6 hrs ¯\\\_(ツ)\_/¯)
3. filter() with loading all data in memory, only a single boolean column (never ends).

Can someone please help?

Below is a sample code for small dataset.

```
from datasets import load_dataset
dataset = load_dataset('glue', 'mrpc', split='train')
dataset = dataset.map(lambda x: {'flag': random.randint(0,1)==1})

def _amplify(data):
        return data

dataset = dataset.filter(_amplify, batch_size=1024, keep_in_memory=False, input_columns=['flag'])
```

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1796/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}