map/filter multiprocessing raises errors and corrupts datasets
It seems that I ran into the same problem
```
def tokenize(cols, example):
  for in_col, out_col in cols.items():
    example[out_col] = hf_tokenizer.convert_tokens_to_ids(hf_tokenizer.tokenize(example[in_col]))
  return example
cola = datasets.load_dataset('glue', 'cola')
tokenized_cola = cola.map(partial(tokenize, {'sentence': 'text_idxs'}),
                             num_proc=2,)
```
and it outpus (exceprts)
```
Concatenating 2 shards from multiprocessing
Set __getitem__(key) output type to python objects for ['idx', 'label', 'sentence', 'text_idxs'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Testing the mapped function outputs
Testing finished, running the mapping function on the dataset
Done writing 532 indices in 4256 bytes .
Done writing 531 indices in 4248 bytes .
Process #0 will write at /home/yisiang/.cache/huggingface/datasets/glue/cola/1.0.0/930e9d141872db65102cabb9fa8ac01c11ffc8a1b72c2e364d8cdda4610df542/tokenized_test_00000_of_00002.arrow
Process #1 will write at /home/yisiang/.cache/huggingface/datasets/glue/cola/1.0.0/930e9d141872db65102cabb9fa8ac01c11ffc8a1b72c2e364d8cdda4610df542/tokenized_test_00001_of_00002.arrow
Spawning 2 processes
```
and then the program never stop.
After upgrading to the 1.0 started seeing errors in my data loading script after enabling multiprocessing.

```python
    ...
    ner_ds_dict = ner_ds.train_test_split(test_size=test_pct, shuffle=True, seed=seed)
    ner_ds_dict["validation"] = ner_ds_dict["test"]
    rel_ds_dict = rel_ds.train_test_split(test_size=test_pct, shuffle=True, seed=seed)
    rel_ds_dict["validation"] = rel_ds_dict["test"]
    return ner_ds_dict, rel_ds_dict
```

The first train_test_split, `ner_ds`/`ner_ds_dict`, returns a `train` and `test` split that are iterable.
The second, `rel_ds`/`rel_ds_dict` in this case, returns a Dataset dict that has rows but if selected from or sliced into into returns an empty dictionary. eg `rel_ds_dict['train'][0] == {}` and `rel_ds_dict['train'][0:100] == {}`.

Ok I think I know the problem -- the rel_ds was mapped though a mapper with `num_proc=12`. If I remove `num_proc`. The dataset loads.

I also see errors with other map and filter functions when `num_proc` is set.

```
Done writing 67 indices in 536 bytes .
Done writing 67 indices in 536 bytes .
Fatal Python error: PyCOND_WAIT(gil_cond) failed
```
map/filter multiprocessing raises errors and corrupts datasets 
 After upgrading to the 1.0 started seeing errors in my data loading script after enabling multiprocessing.

```python
    ...
    ner_ds_dict = ner_ds.train_test_split(test_size=test_pct, shuffle=True, seed=seed)
    ner_ds_dict["validation"] = ner_ds_dict["test"]
    rel_ds_dict = rel_ds.train_test_split(test_size=test_pct, shuffle=True, seed=seed)
    rel_ds_dict["validation"] = rel_ds_dict["test"]
    return ner_ds_dict, rel_ds_dict
```

The first train_test_split, `ner_ds`/`ner_ds_dict`, returns a `train` and `test` split that are iterable.
The second, `rel_ds`/`rel_ds_dict` in this case, returns a Dataset dict that has rows but if selected from or sliced into into returns an empty dictionary. eg `rel_ds_dict['train'][0] == {}` and `rel_ds_dict['train'][0:100] == {}`.

Ok I think I know the problem -- the rel_ds was mapped though a mapper with `num_proc=12`. If I remove `num_proc`. The dataset loads.

I also see errors with other map and filter functions when `num_proc` is set.

```
Done writing 67 indices in 536 bytes .
Done writing 67 indices in 536 bytes .
Fatal Python error: PyCOND_WAIT(gil_cond) failed
``` 
 It seems that I ran into the same problem
```
def tokenize(cols, example):
  for in_col, out_col in cols.items():
    example[out_col] = hf_tokenizer.convert_tokens_to_ids(hf_tokenizer.tokenize(example[in_col]))
  return example
cola = datasets.load_dataset('glue', 'cola')
tokenized_cola = cola.map(partial(tokenize, {'sentence': 'text_idxs'}),
                             num_proc=2,)
```
and it outpus (exceprts)
```
Concatenating 2 shards from multiprocessing
Set __getitem__(key) output type to python objects for ['idx', 'label', 'sentence', 'text_idxs'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Testing the mapped function outputs
Testing finished, running the mapping function on the dataset
Done writing 532 indices in 4256 bytes .
Done writing 531 indices in 4248 bytes .
Process #0 will write at /home/yisiang/.cache/huggingface/datasets/glue/cola/1.0.0/930e9d141872db65102cabb9fa8ac01c11ffc8a1b72c2e364d8cdda4610df542/tokenized_test_00000_of_00002.arrow
Process #1 will write at /home/yisiang/.cache/huggingface/datasets/glue/cola/1.0.0/930e9d141872db65102cabb9fa8ac01c11ffc8a1b72c2e364d8cdda4610df542/tokenized_test_00001_of_00002.arrow
Spawning 2 processes
```
and then the program never stop.