pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
{'login': 'conceptofmind', 'id': 25208228, 'node_id': 'MDQ6VXNlcjI1MjA4MjI4', 'avatar_url': 'https://avatars.githubusercontent.com/u/25208228?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/conceptofmind', 'html_url': 'https://github.com/conceptofmind', 'followers_url': 'https://api.github.com/users/conceptofmind/followers', 'following_url': 'https://api.github.com/users/conceptofmind/following{/other_user}', 'gists_url': 'https://api.github.com/users/conceptofmind/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/conceptofmind/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/conceptofmind/subscriptions', 'organizations_url': 'https://api.github.com/users/conceptofmind/orgs', 'repos_url': 'https://api.github.com/users/conceptofmind/repos', 'events_url': 'https://api.github.com/users/conceptofmind/events{/privacy}', 'received_events_url': 'https://api.github.com/users/conceptofmind/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892857, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODU3', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/bug', 'name': 'bug', 'color': 'd73a4a', 'default': True, 'description': "Something isn't working"}]
[]
['Thanks for reporting @conceptofmind.\r\n\r\nCould you please give details about your environment? \r\n```\r\n## Environment info\r\n<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->\r\n- `datasets` version:\r\n- Platform:\r\n- Python version:\r\n- PyArrow version:\r\n```'
 'Hi @albertvillanova ,\r\n\r\nHere is the environment information:\r\n```\r\n- `datasets` version: 2.3.2\r\n- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.27\r\n- Python version: 3.9.12\r\n- PyArrow version: 7.0.0\r\n- Pandas version: 1.4.2\r\n```\r\nThanks,\r\n\r\nEnrico'
 'I think this issue is solved here https://discuss.huggingface.co/t/minhash-deduplication/19992/12?u=loubnabnl, this only happens for very large datasets we will update it in CodeParrot code'
 'Hi @loubnabnl,\r\n\r\nYes, the issue is solved in the discussion thread.\r\n\r\nI will close this issue.\r\n\r\nThank you again for all of your help.\r\n\r\nEnrico']
## Describe the bug
Following the example in CodeParrot, I receive an array size limitation error when deduplicating larger datasets.

## Steps to reproduce the bug
```python
dataset_name = "the_pile"
ds = load_dataset(dataset_name, split="train")
ds = ds.map(preprocess, num_proc=num_workers)
uniques = set(ds.unique("hash"))
```
Gists for minimum reproducible example:
https://gist.github.com/conceptofmind/c5804428ea1bd89767815f9cd5f02d9a
https://gist.github.com/conceptofmind/feafb07e236f28d79c2d4b28ffbdb6e2

## Expected results
Chunking and writing out a deduplicated dataset. 

## Actual results
```
return dataset._data.column(column).unique().to_pylist()
File "pyarrow/table.pxi", line 394, in pyarrow.lib.ChunkedArray.unique
File "pyarrow/_compute.pyx", line 531, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 330, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 124, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4782/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}