Datasets' cache not re-used
## Describe the bug
For most tokenizers I have tested (e.g. the RoBERTa tokenizer), the data preprocessing cache are not fully reused in the first few runs, although their `.arrow` cache files are in the cache directory.

## Steps to reproduce the bug
Here is a reproducer. The GPT2 tokenizer works perfectly with caching, but not the RoBERTa tokenizer in this example.
```python
from datasets import load_dataset
from transformers import AutoTokenizer

raw_datasets = load_dataset("wikitext", "wikitext-2-raw-v1")
# tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
text_column_name = "text"
column_names = raw_datasets["train"].column_names

def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

tokenized_datasets = raw_datasets.map(
    tokenize_function,
    batched=True,
    remove_columns=column_names,
    load_from_cache_file=True,
    desc="Running tokenizer on every text in dataset",
)
```

## Expected results
No tokenization would be required after the 1st run. Everything should be loaded from the cache.

## Actual results
Tokenization for some subsets are repeated at the 2nd and 3rd run. Starting from the 4th run, everything are loaded from cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 18.04.6 LTS
- Python version: 3.6.9
- PyArrow version: 6.0.1

diff --git a/src/datasets/dataset_dict.py b/src/datasets/dataset_dict.py
index 1df4527ca..240d031e1 100644
--- a/src/datasets/dataset_dict.py
+++ b/src/datasets/dataset_dict.py
@@ -418,30 +418,29 @@ class DatasetDict(dict):
         self._check_values_type()
         if cache_file_names is None:
             cache_file_names = {k: None for k in self}
-        return DatasetDict(
-            {
-                k: dataset.map(
-                    function=function,
-                    with_indices=with_indices,
-                    with_rank=with_rank,
-                    input_columns=input_columns,
-                    batched=batched,
-                    batch_size=batch_size,
-                    drop_last_batch=drop_last_batch,
-                    remove_columns=remove_columns,
-                    keep_in_memory=keep_in_memory,
-                    load_from_cache_file=load_from_cache_file,
-                    cache_file_name=cache_file_names[k],
-                    writer_batch_size=writer_batch_size,
-                    features=features,
-                    disable_nullable=disable_nullable,
-                    fn_kwargs=fn_kwargs,
-                    num_proc=num_proc,
-                    desc=desc,
-                )
-                for k, dataset in self.items()
-            }
-        )
+        out = {
+            k: self[k].map(
+                function=function,
+                with_indices=with_indices,
+                with_rank=with_rank,
+                input_columns=input_columns,
+                batched=batched,
+                batch_size=batch_size,
+                drop_last_batch=drop_last_batch,
+                remove_columns=remove_columns,
+                keep_in_memory=keep_in_memory,
+                load_from_cache_file=load_from_cache_file,
+                cache_file_name=cache_file_names[k],
+                writer_batch_size=writer_batch_size,
+                features=features,
+                disable_nullable=disable_nullable,
+                fn_kwargs=fn_kwargs,
+                num_proc=num_proc,
+                desc=desc,
+            )
+            for k in sorted(self)  # deterministic split order for https://github.com/huggingface/datasets/issues/3847
+        }
+        return DatasetDict({k: out[k] for k in self})
 
     def filter(
         self,
@@ -450,7 +449,6 @@ class DatasetDict(dict):
         input_columns: Optional[Union[str, List[str]]] = None,
         batched: bool = False,
         batch_size: Optional[int] = 1000,
-        remove_columns: Optional[List[str]] = None,
         keep_in_memory: bool = False,
         load_from_cache_file: bool = True,
         cache_file_names: Optional[Dict[str, Optional[str]]] = None,
@@ -492,25 +490,24 @@ class DatasetDict(dict):
         self._check_values_type()
         if cache_file_names is None:
             cache_file_names = {k: None for k in self}
-        return DatasetDict(
-            {
-                k: dataset.filter(
-                    function=function,
-                    with_indices=with_indices,
-                    input_columns=input_columns,
-                    batched=batched,
-                    batch_size=batch_size,
-                    keep_in_memory=keep_in_memory,
-                    load_from_cache_file=load_from_cache_file,
-                    cache_file_name=cache_file_names[k],
-                    writer_batch_size=writer_batch_size,
-                    fn_kwargs=fn_kwargs,
-                    num_proc=num_proc,
-                    desc=desc,
-                )
-                for k, dataset in self.items()
-            }
-        )
+        out = {
+            k: self[k].filter(
+                function=function,
+                with_indices=with_indices,
+                input_columns=input_columns,
+                batched=batched,
+                batch_size=batch_size,
+                keep_in_memory=keep_in_memory,
+                load_from_cache_file=load_from_cache_file,
+                cache_file_name=cache_file_names[k],
+                writer_batch_size=writer_batch_size,
+                fn_kwargs=fn_kwargs,
+                num_proc=num_proc,
+                desc=desc,
+            )
+            for k in sorted(self)  # deterministic split order for https://github.com/huggingface/datasets/issues/3847
+        }
+        return DatasetDict({k: out[k] for k in self})
 
     def sort(
         self,
['src/datasets/dataset_dict.py']