[arrow_dataset.py] add new filter function
{'login': 'patrickvonplaten', 'id': 23423619, 'node_id': 'MDQ6VXNlcjIzNDIzNjE5', 'avatar_url': 'https://avatars.githubusercontent.com/u/23423619?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/patrickvonplaten', 'html_url': 'https://github.com/patrickvonplaten', 'followers_url': 'https://api.github.com/users/patrickvonplaten/followers', 'following_url': 'https://api.github.com/users/patrickvonplaten/following{/other_user}', 'gists_url': 'https://api.github.com/users/patrickvonplaten/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/patrickvonplaten/subscriptions', 'organizations_url': 'https://api.github.com/users/patrickvonplaten/orgs', 'repos_url': 'https://api.github.com/users/patrickvonplaten/repos', 'events_url': 'https://api.github.com/users/patrickvonplaten/events{/privacy}', 'received_events_url': 'https://api.github.com/users/patrickvonplaten/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["I agree that a `.filter` method would be VERY useful and appreciated. I'm not a big fan of using `flatten_nested` as it completely breaks down the structure of the example and it may create bugs. Right now I think it may not work for nested structures. Maybe there's a simpler way that we've not figured out yet."
 'Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:\r\n```python\r\nfor i in range(num_examples):\r\n    example = map_nested(lambda x: x[i], batch)\r\n    # ... then test to keep it or not\r\n```'
 "> Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:\r\n> \r\n> ```python\r\n> for i in range(num_examples):\r\n>     example = map_nested(lambda x: x[i], batch)\r\n>     # ... then test to keep it or not\r\n> ```\r\n\r\nAwesome I'll check it out :-) "
 '> Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:\r\n> \r\n> ```python\r\n> for i in range(num_examples):\r\n>     example = map_nested(lambda x: x[i], batch)\r\n>     # ... then test to keep it or not\r\n> ```\r\n\r\nAwesome this function is definitely much nicer!'
 'Actually I just realized that `map_nested` might not work either as it applies the function at the very last list of the structure. However we can imagine that a single example has also a list in its structure:\r\n```python\r\none_example = {\r\n    "title": "blabla",\r\n    "paragraphs": [\r\n        "p1", "p2", ...\r\n    ]\r\n}\r\n```'
 "We'll probably have to take into account the `dset._data.schema` to extract the examples from the batch."
 '> Actually I just realized that `map_nested` might not work either as it applies the function at the very last list of the structure. However we can imagine that a single example has also a list in its structure:\r\n> \r\n> ```python\r\n> one_example = {\r\n>     "title": "blabla",\r\n>     "paragraphs": [\r\n>         "p1", "p2", ...\r\n>     ]\r\n> }\r\n> ```\r\n\r\nThey both work. I\'m using it on trivia_qa which is pretty nested. If you use the option `dict_only=True` I think it\'s fine.'
 "> We'll probably have to take into account the `dset._data.schema` to extract the examples from the batch.\r\n\r\nWhy? "
 "Actually it's fine. I guess this is going to be yet another thing to be unit-tested just to make sure ^^"
 'Yes, I will need to add tests and documentation! \r\n@thomwolf - would a function like this be ok? It abstracts `.map()` a bit which might be hard to understand. '
 'I tried on some datasets with nested structure and it works fine ! Great work :D \r\n'
 'Awesome :-), I will add documentation and some simple unittests'
 'Ok merging!']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/214', 'html_url': 'https://github.com/huggingface/datasets/pull/214', 'diff_url': 'https://github.com/huggingface/datasets/pull/214.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/214.patch', 'merged_at': datetime.datetime(2020, 5, 29, 11, 32, 20)}
The `.map()` function is super useful, but can IMO a bit tedious when filtering certain examples.
I think, filtering out examples is also a very common operation people would like to perform on datasets.

This PR is a proposal to add a `.filter()` function in the same spirit than the `.map()` function.

Here is a sample code you can play around with:

```python
ds = nlp.load_dataset("squad", split="validation[:10%]")


def remove_under_idx_5(example, idx):
    return idx < 5


def only_keep_examples_with_is_in_context(example):
    return "is" in example["context"]


result_keep_only_first_5 = ds.filter(remove_under_idx_5, with_indices=True, load_from_cache_file=False)
result_keep_examples_with_is_in_context = ds.filter(only_keep_examples_with_is_in_context, load_from_cache_file=False)

print("Original number of examples: {}".format(len(ds)))
print("First five examples number of examples: {}".format(len(result_keep_only_first_5)))
print("Is in context examples number of examples: {}".format(len(result_keep_examples_with_is_in_context)))
```
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/214/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}