Very slow data loading on large dataset
@agemagician you can give a try at a multithreaded version if you want (currently on the #548).

To test it, you just need to copy the new `text` processing script which is [here](https://github.com/huggingface/nlp/blob/07d92a82b7594498ff702f3cca55c074e2052257/datasets/text/text.py) somewhere on your drive and give it's local path instead of `text` to `load_dataset`. E.g. in your example:
```python
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('./datasets/text.py',   # path to where you've dowloaded the multi-threaded text loading script
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```
I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ?
Very slow data loading on large dataset 
 I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ? 
 @agemagician you can give a try at a multithreaded version if you want (currently on the #548).

To test it, you just need to copy the new `text` processing script which is [here](https://github.com/huggingface/nlp/blob/07d92a82b7594498ff702f3cca55c074e2052257/datasets/text/text.py) somewhere on your drive and give it's local path instead of `text` to `load_dataset`. E.g. in your example:
```python
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('./datasets/text.py',   # path to where you've dowloaded the multi-threaded text loading script
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```