Saving processed dataset running infinitely
am suspicious of this thing? what's the purpose of this? pickling and unplickling
`self = pickle.loads(pickle.dumps(self))`

```
    def save_to_disk(self, dataset_path: str, fs=None):
        """
        Saves a dataset to a dataset directory, or in a filesystem using either :class:`datasets.filesystem.S3FileSystem` or any implementation of ``fsspec.spec.AbstractFileSystem``.

        Args:
            dataset_path (``str``): path (e.g. ``dataset/train``) or remote uri (e.g. ``s3://my-bucket/dataset/train``) of the dataset directory where the dataset will be saved to
            fs (Optional[:class:`datasets.filesystem.S3FileSystem`,``fsspec.spec.AbstractFileSystem``],  `optional`, defaults ``None``): instance of :class:`datasets.filesystem.S3FileSystem` or ``fsspec.spec.AbstractFileSystem`` used to download the files from remote filesystem.
        """
        assert (
            not self.list_indexes()
        ), "please remove all the indexes using `dataset.drop_index` before saving a dataset"
        self = pickle.loads(pickle.dumps(self))
  ```
I have a text dataset of size 220M.

For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 3hrs. I used map() with batch size 1024 and multi-process with 96 processes.

filter() function was way to slow, so I used a hack to use pyarrow filter table function, which is damm fast. Mentioned [here](https://github.com/huggingface/datasets/issues/1796)

```dataset._data = dataset._data.filter(...)```
It took 1 hr for the filter.

Then i use `save_to_disk()` on processed dataset and it is running forever.

I have been waiting since 8 hrs, it has not written a single byte. 

Infact it has actually read from disk more than 100GB, screenshot below shows the stats using `iotop`. 
Second process is the one.
<img width="1672" alt="Screenshot 2021-02-19 at 6 36 53 PM" src="https://user-images.githubusercontent.com/20911334/108508197-7325d780-72e1-11eb-8369-7c057d137d81.png">


I am not able to figure out, whether this is some issue with dataset library or that it is due to my hack for filter() function.
Saving processed dataset running infinitely 
 I have a text dataset of size 220M.

For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 3hrs. I used map() with batch size 1024 and multi-process with 96 processes.

filter() function was way to slow, so I used a hack to use pyarrow filter table function, which is damm fast. Mentioned [here](https://github.com/huggingface/datasets/issues/1796)

```dataset._data = dataset._data.filter(...)```
It took 1 hr for the filter.

Then i use `save_to_disk()` on processed dataset and it is running forever.

I have been waiting since 8 hrs, it has not written a single byte. 

Infact it has actually read from disk more than 100GB, screenshot below shows the stats using `iotop`. 
Second process is the one.
<img width="1672" alt="Screenshot 2021-02-19 at 6 36 53 PM" src="https://user-images.githubusercontent.com/20911334/108508197-7325d780-72e1-11eb-8369-7c057d137d81.png">


I am not able to figure out, whether this is some issue with dataset library or that it is due to my hack for filter() function. 
 am suspicious of this thing? what's the purpose of this? pickling and unplickling
`self = pickle.loads(pickle.dumps(self))`

```
    def save_to_disk(self, dataset_path: str, fs=None):
        """
        Saves a dataset to a dataset directory, or in a filesystem using either :class:`datasets.filesystem.S3FileSystem` or any implementation of ``fsspec.spec.AbstractFileSystem``.

        Args:
            dataset_path (``str``): path (e.g. ``dataset/train``) or remote uri (e.g. ``s3://my-bucket/dataset/train``) of the dataset directory where the dataset will be saved to
            fs (Optional[:class:`datasets.filesystem.S3FileSystem`,``fsspec.spec.AbstractFileSystem``],  `optional`, defaults ``None``): instance of :class:`datasets.filesystem.S3FileSystem` or ``fsspec.spec.AbstractFileSystem`` used to download the files from remote filesystem.
        """
        assert (
            not self.list_indexes()
        ), "please remove all the indexes using `dataset.drop_index` before saving a dataset"
        self = pickle.loads(pickle.dumps(self))
  ```