Faster Shuffling?
{'login': 'mitchellgordon95', 'id': 7490438, 'node_id': 'MDQ6VXNlcjc0OTA0Mzg=', 'avatar_url': 'https://avatars.githubusercontent.com/u/7490438?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/mitchellgordon95', 'html_url': 'https://github.com/mitchellgordon95', 'followers_url': 'https://api.github.com/users/mitchellgordon95/followers', 'following_url': 'https://api.github.com/users/mitchellgordon95/following{/other_user}', 'gists_url': 'https://api.github.com/users/mitchellgordon95/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/mitchellgordon95/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/mitchellgordon95/subscriptions', 'organizations_url': 'https://api.github.com/users/mitchellgordon95/orgs', 'repos_url': 'https://api.github.com/users/mitchellgordon95/repos', 'events_url': 'https://api.github.com/users/mitchellgordon95/events{/privacy}', 'received_events_url': 'https://api.github.com/users/mitchellgordon95/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['I think the slowness here probably come from the fact that we are copying from and to python.\r\n\r\n@lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?'
 "> @lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?\r\n\r\nI just tried with `writer.write_table` with tables of 1000 elements and it's slower that the solution in #405 \r\n\r\nOn my side (select 10 000 examples):\r\n- Original implementation: 12s\r\n- Batched solution: 100ms\r\n- solution using arrow tables: 350ms\r\n\r\nI'll try with arrays and record batches to see if we can make it work."
 'I tried using `.take` from pyarrow recordbatches but it doesn\'t improve the speed that much:\r\n```python\r\nimport nlp\r\nimport numpy as np\r\n\r\ndset = nlp.Dataset.from_file("dummy_test_select.arrow")  # dummy dataset with 100000 examples like {"a": "h"*512}\r\nindices = np.random.randint(0, 100_000, 1000_000)\r\n```\r\n\r\n```python\r\n%%time\r\nbatch_size = 10_000\r\nwriter = ArrowWriter(schema=dset.schema, path="dummy_path",\r\n                     writer_batch_size=1000, disable_nullable=False)\r\nfor i in tqdm(range(0, len(indices), batch_size)):\r\n    table = pa.concat_tables(dset._data.slice(int(i), 1) for i in indices[i : min(len(indices), i + batch_size)])\r\n    batch = table.to_pydict()\r\n    writer.write_batch(batch)\r\nwriter.finalize()\r\n# 9.12s\r\n```\r\n\r\n\r\n```python\r\n%%time\r\nbatch_size = 10_000\r\nwriter = ArrowWriter(schema=dset.schema, path="dummy_path", \r\n                     writer_batch_size=1000, disable_nullable=False)\r\nfor i in tqdm(range(0, len(indices), batch_size)):\r\n    batch_indices = indices[i : min(len(indices), i + batch_size)]\r\n    # First, extract only the indices that we need with a mask\r\n    mask = [False] * len(dset)\r\n    for k in batch_indices:\r\n        mask[k] = True\r\n    t_batch = dset._data.filter(pa.array(mask))\r\n    # Second, build the list of indices for the filtered table, and taking care of duplicates\r\n    rev_positions = {}\r\n    duplicates = 0\r\n    for i, j in enumerate(sorted(batch_indices)):\r\n        if j in rev_positions:\r\n            duplicates += 1\r\n        else:\r\n            rev_positions[j] = i - duplicates\r\n    rev_map = [rev_positions[j] for j in batch_indices]\r\n    # Third, use `.take` from the combined recordbatch\r\n    t_combined = t_batch.combine_chunks()  # load in memory\r\n    recordbatch = t_combined.to_batches()[0]\r\n    table = pa.Table.from_arrays(\r\n        [recordbatch[c].take(pa.array(rev_map)) for c in range(len(dset._data.column_names))],\r\n        schema=writer.schema\r\n    )\r\n    writer.write_table(table)\r\nwriter.finalize()\r\n# 3.2s\r\n```\r\n'
 'Shuffling is now significantly faster thanks to #513 \r\nFeel free to play with it now :)\r\n\r\nClosing this one, but feel free to re-open if you have other questions']
Consider shuffling bookcorpus:

```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (even with the faster version of select from #405). I've also tried with `keep_in_memory=True` and `writer_batch_size=1000`.

But I can also just write the lines to a text file:

```
batch_size = 100000
with open('tmp.txt', 'w+') as out_f:
    for i in tqdm(range(0, len(dataset), batch_size)):
        batch = dataset[i:i+batch_size]['text']
        print("\n".join(batch), file=out_f)
```

Which completes in a couple minutes, followed by `shuf tmp.txt > tmp2.txt` which completes in under a minute. And finally,

```
dataset = nlp.load_dataset('text', data_files='tmp2.txt')
```

Which completes in under 10 minutes. I read up on Apache Arrow this morning, and it seems like the columnar data format is not especially well-suited to shuffling rows, since moving items around requires a lot of book-keeping. 

Is shuffle inherently slow, or am I just using it wrong? And if it is slow, would it make sense to try converting the data to a row-based format on disk and then shuffling? (Instead of calling select with a random permutation, as is currently done.)
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/406/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}