Add option to delete temporary files (e.g. extracted files) when loading dataset
Hi !
If we want something more general, we could either
1. delete the extracted files after the arrow data generation automatically, or 
2. delete each extracted file during the arrow generation right after it has been closed.

Solution 2 is better to save disk space during the arrow generation. Is it what you had in mind ?

The API could look like
```python
load_dataset(..., delete_extracted_files_after_usage=True)
```

In terms of implementation, here are some directions we could take for each solution:
1. get the list of the extracted files from the DownloadManager and then delete them after the dataset is processed. This can be implemented in `download_and_prepare` I guess
2. maybe wrap and mock `open` in the builder to make it delete the file when the file is closed.
I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution.
Add option to delete temporary files (e.g. extracted files) when loading dataset 
 I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution. 
 Hi !
If we want something more general, we could either
1. delete the extracted files after the arrow data generation automatically, or 
2. delete each extracted file during the arrow generation right after it has been closed.

Solution 2 is better to save disk space during the arrow generation. Is it what you had in mind ?

The API could look like
```python
load_dataset(..., delete_extracted_files_after_usage=True)
```

In terms of implementation, here are some directions we could take for each solution:
1. get the list of the extracted files from the DownloadManager and then delete them after the dataset is processed. This can be implemented in `download_and_prepare` I guess
2. maybe wrap and mock `open` in the builder to make it delete the file when the file is closed.