[speedup] Use indices mappings instead of deepcopy for all the samples reordering methods
{'login': 'thomwolf', 'id': 7353373, 'node_id': 'MDQ6VXNlcjczNTMzNzM=', 'avatar_url': 'https://avatars.githubusercontent.com/u/7353373?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/thomwolf', 'html_url': 'https://github.com/thomwolf', 'followers_url': 'https://api.github.com/users/thomwolf/followers', 'following_url': 'https://api.github.com/users/thomwolf/following{/other_user}', 'gists_url': 'https://api.github.com/users/thomwolf/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/thomwolf/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/thomwolf/subscriptions', 'organizations_url': 'https://api.github.com/users/thomwolf/orgs', 'repos_url': 'https://api.github.com/users/thomwolf/repos', 'events_url': 'https://api.github.com/users/thomwolf/events{/privacy}', 'received_events_url': 'https://api.github.com/users/thomwolf/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Ok I fixed `concatenate_datasets` and added tests
Feel free to merge if it's good for you @thomwolf  Ok, adding some benchmarks for map/filters and then I'll merge Warning from pytorch that we should maybe consider at some point @lhoestq:
```
/__w/nlp/nlp/src/nlp/arrow_dataset.py:648: UserWarning: The given NumPy array is not writeable,
and PyTorch does not support non-writeable tensors. This means you can write to the underlying
(supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to
protect its data or make it writeable before converting it to a tensor. This type of warning will be
suppressed for the rest of this program.
(Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
532
  return torch.tensor(x, **format_kwargs)
``` > Warning from pytorch that we should maybe consider at some point @lhoestq:
> 
> ```
> /__w/nlp/nlp/src/nlp/arrow_dataset.py:648: UserWarning: The given NumPy array is not writeable,
> and PyTorch does not support non-writeable tensors. This means you can write to the underlying
> (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to
> protect its data or make it writeable before converting it to a tensor. This type of warning will be
> suppressed for the rest of this program.
> (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
> 532
>   return torch.tensor(x, **format_kwargs)
> ```

Not sure why we have that, it's probably linked to zero copy from arrow to numpy
Use an indices mapping instead of rewriting the dataset for all the samples re-ordering/selection methods (`select`, `sort`, `shuffle`, `shard`, `train_test_split`).

Added a `flatten_indices` method which copy the dataset to a new table to remove the indices mapping with tests.

All the samples re-ordering/selection methods should be a lot faster. The downside is that iterating on very large batch of the dataset might be a little slower when we have changed the order of the samples since with in these case we use `pyarrow.Table.take` instead of `pyarrow.Table.slice`. There is no free lunch but the speed of iterating over the dataset is rarely the bottleneck.

*Backward breaking change*: the `cache_file_name` argument in all the samples re-ordering/selection methods (`select`, `sort`, `shuffle`, `shard`, `train_test_split`) is now called `indices_cache_file_name` on purpose to make it explicit to the user that this caching file is used for caching the indices mapping and not the dataset itself.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/513/reactions', 'total_count': 1, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 1, 'eyes': 0}
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/513', 'html_url': 'https://github.com/huggingface/datasets/pull/513', 'diff_url': 'https://github.com/huggingface/datasets/pull/513.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/513.patch', 'merged_at': '2020-08-28T08:41:50Z'}