Add option to delete temporary files (e.g. extracted files) when loading dataset
Ok yes. I think this is an important feature to be able to use large datasets which are pretty much always compressed files.

In particular now this requires to keep the extracted file on the drive if you want to avoid reprocessing the dataset so in my case, this require using always ~400GB of drive instead of just 200GB (which is already significant). 

Two nice features would be to:
- allow to delete the extracted files without loosing the ability to load the dataset from the cached arrow-file
- streamlined decompression when only the currently read file is extracted - this might require to read the list of files from the extracted archives before processing them?
I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution.
Add option to delete temporary files (e.g. extracted files) when loading dataset 
 I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution. 
 Ok yes. I think this is an important feature to be able to use large datasets which are pretty much always compressed files.

In particular now this requires to keep the extracted file on the drive if you want to avoid reprocessing the dataset so in my case, this require using always ~400GB of drive instead of just 200GB (which is already significant). 

Two nice features would be to:
- allow to delete the extracted files without loosing the ability to load the dataset from the cached arrow-file
- streamlined decompression when only the currently read file is extracted - this might require to read the list of files from the extracted archives before processing them?