ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk
{'login': 'simonschoe', 'id': 53626067, 'node_id': 'MDQ6VXNlcjUzNjI2MDY3', 'avatar_url': 'https://avatars.githubusercontent.com/u/53626067?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/simonschoe', 'html_url': 'https://github.com/simonschoe', 'followers_url': 'https://api.github.com/users/simonschoe/followers', 'following_url': 'https://api.github.com/users/simonschoe/following{/other_user}', 'gists_url': 'https://api.github.com/users/simonschoe/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/simonschoe/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/simonschoe/subscriptions', 'organizations_url': 'https://api.github.com/users/simonschoe/orgs', 'repos_url': 'https://api.github.com/users/simonschoe/repos', 'events_url': 'https://api.github.com/users/simonschoe/events{/privacy}', 'received_events_url': 'https://api.github.com/users/simonschoe/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["Hi ! To help me understand the situation, can you print the values of  `load_from_disk(PATH_DATA_CLS_A)['train']._indices_data_files` and `load_from_disk(PATH_DATA_CLS_B)['train']._indices_data_files` ?\r\nThey should both have a path to an arrow file\r\n\r\nAlso note that from #2025 concatenating datasets will no longer have such restrictions."
 "Sure, thanks for the fast reply!\r\n\r\nFor dataset A: `[{'filename': 'drive/MyDrive/data_target_task/dataset_a/train/cache-4797266bf4db1eb7.arrow'}]`\r\nFor dataset B: `[]`\r\n\r\nNo clue why for B it returns nothing. `PATH_DATA_CLS_B` is exactly the same in `save_to_disk` and `load_from_disk`... Also I can verify that the folder physically exists under 'drive/MyDrive/data_target_task/dataset_b/'"
 "In the next release you'll be able to concatenate any kinds of dataset (either from memory or from disk).\r\n\r\nFor now I'd suggest you to flatten the indices of the A and B datasets. This will remove the indices mapping and you will be able to concatenate them. You can flatten the indices with\r\n```python\r\ndataset = dataset.flatten_indices()\r\n```"
 'Indeed this works. Not the most elegant solution, but it does the trick. Thanks a lot! ']
Hi there,

I am trying to concat two datasets that I've previously saved to disk via `save_to_disk()` like so (note that both are saved as `DataDict`, `PATH_DATA_CLS_*` are `Path`-objects):
```python
concatenate_datasets([load_from_disk(PATH_DATA_CLS_A)['train'], load_from_disk(PATH_DATA_CLS_B)['train']])
```
Yielding the following error:
```python
ValueError: Datasets' indices should ALL come from memory, or should ALL come from disk.
However datasets' indices [1] come from memory and datasets' indices [0] come from disk.
```
Been trying to solve this for quite some time now. Both `DataDict` have been created by reading in a `csv` via `load_dataset` and subsequently processed using the various `datasets` methods (i.e. filter, map, remove col, rename col). Can't figure out tho...

`load_from_disk(PATH_DATA_CLS_A)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 785
})
```
`load_from_disk(PATH_DATA_CLS_B)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 3341
})
```
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2040/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}