Error when concatenate_datasets
> As you can see in the error the test checks if `indices_mappings_in_memory` is True or not, which is different from the test you do in your script. In a dataset, both the data and the indices mapping can be either on disk or in memory.
> 
> The indices mapping correspond to a mapping on top of the data table that is used to re-order/select a sample of the original data table. For example if you do `dataset.train_test_split`, then the resulting train and test datasets will have both an indices mapping to tell which examples are in train and which ones in test.
> 
> Before saving your datasets on disk, you should call `dataset.flatten_indices()` to remove the indices mapping. It should fix your issue. Under the hood it will create a new data table using the indices mapping. The new data table is going to be a subset of the old one (for example taking only the test set examples), and since the indices mapping will be gone you'll be able to concatenate your datasets.

`dataset.flatten_indices()` solved my problem, thanks so much!
Hello, when I concatenate two dataset loading  from disk, I encountered a problem:
```
test_dataset = load_from_disk('data/test_dataset')
trn_dataset = load_from_disk('data/train_dataset')

train_dataset = concatenate_datasets([trn_dataset, test_dataset])
```
And it reported ValueError blow:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-38-74fa525512ca> in <module>
----> 1 train_dataset = concatenate_datasets([trn_dataset, test_dataset])

/opt/miniconda3/lib/python3.7/site-packages/datasets/arrow_dataset.py in concatenate_datasets(dsets, info, split)
   2547                 "However datasets' indices {} come from memory and datasets' indices {} come from disk.".format(
   2548                     [i for i in range(len(dsets)) if indices_mappings_in_memory[i]],
-> 2549                     [i for i in range(len(dsets)) if not indices_mappings_in_memory[i]],
   2550                 )
   2551             )

ValueError: Datasets' indices should ALL come from memory, or should ALL come from disk.
However datasets' indices [1] come from memory and datasets' indices [0] come from disk.
```

But it's curious both of my datasets loading from disk, so I check the source code in `arrow_dataset.py` about the Error:
```
trn_dataset._data_files
# output
[{'filename': 'data/train_dataset/csv-train.arrow', 'skip': 0, 'take': 593264}]

test_dataset._data_files
# output
[{'filename': 'data/test_dataset/csv-test.arrow', 'skip': 0, 'take': 424383}]

print([not dset._data_files for dset in [trn_dataset, test_dataset]])
# [False, False]

# And I tested the code the same as arrow_dataset, but nothing happened
dsets = [trn_dataset, test_dataset]
dsets_in_memory = [not dset._data_files for dset in dsets]
if any(dset_in_memory != dsets_in_memory[0] for dset_in_memory in dsets_in_memory):
    raise ValueError(
        "Datasets should ALL come from memory, or should ALL come from disk.\n"
        "However datasets {} come from memory and datasets {} come from disk.".format(
            [i for i in range(len(dsets)) if dsets_in_memory[i]],
            [i for i in range(len(dsets)) if not dsets_in_memory[i]],
        )
    )
```

Any suggestions would be greatly appreciated! 
Thanks!
Error when concatenate_datasets 
 Hello, when I concatenate two dataset loading  from disk, I encountered a problem:
```
test_dataset = load_from_disk('data/test_dataset')
trn_dataset = load_from_disk('data/train_dataset')

train_dataset = concatenate_datasets([trn_dataset, test_dataset])
```
And it reported ValueError blow:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-38-74fa525512ca> in <module>
----> 1 train_dataset = concatenate_datasets([trn_dataset, test_dataset])

/opt/miniconda3/lib/python3.7/site-packages/datasets/arrow_dataset.py in concatenate_datasets(dsets, info, split)
   2547                 "However datasets' indices {} come from memory and datasets' indices {} come from disk.".format(
   2548                     [i for i in range(len(dsets)) if indices_mappings_in_memory[i]],
-> 2549                     [i for i in range(len(dsets)) if not indices_mappings_in_memory[i]],
   2550                 )
   2551             )

ValueError: Datasets' indices should ALL come from memory, or should ALL come from disk.
However datasets' indices [1] come from memory and datasets' indices [0] come from disk.
```

But it's curious both of my datasets loading from disk, so I check the source code in `arrow_dataset.py` about the Error:
```
trn_dataset._data_files
# output
[{'filename': 'data/train_dataset/csv-train.arrow', 'skip': 0, 'take': 593264}]

test_dataset._data_files
# output
[{'filename': 'data/test_dataset/csv-test.arrow', 'skip': 0, 'take': 424383}]

print([not dset._data_files for dset in [trn_dataset, test_dataset]])
# [False, False]

# And I tested the code the same as arrow_dataset, but nothing happened
dsets = [trn_dataset, test_dataset]
dsets_in_memory = [not dset._data_files for dset in dsets]
if any(dset_in_memory != dsets_in_memory[0] for dset_in_memory in dsets_in_memory):
    raise ValueError(
        "Datasets should ALL come from memory, or should ALL come from disk.\n"
        "However datasets {} come from memory and datasets {} come from disk.".format(
            [i for i in range(len(dsets)) if dsets_in_memory[i]],
            [i for i in range(len(dsets)) if not dsets_in_memory[i]],
        )
    )
```

Any suggestions would be greatly appreciated! 
Thanks! 
 > As you can see in the error the test checks if `indices_mappings_in_memory` is True or not, which is different from the test you do in your script. In a dataset, both the data and the indices mapping can be either on disk or in memory.
> 
> The indices mapping correspond to a mapping on top of the data table that is used to re-order/select a sample of the original data table. For example if you do `dataset.train_test_split`, then the resulting train and test datasets will have both an indices mapping to tell which examples are in train and which ones in test.
> 
> Before saving your datasets on disk, you should call `dataset.flatten_indices()` to remove the indices mapping. It should fix your issue. Under the hood it will create a new data table using the indices mapping. The new data table is going to be a subset of the old one (for example taking only the test set examples), and since the indices mapping will be gone you'll be able to concatenate your datasets.

`dataset.flatten_indices()` solved my problem, thanks so much!