Slow dataloading with big datasets issue persists
@lhoestq and @hwijeen

Despite upgrading to datasets 1.6.2, still experiencing extremely slow (2h00) loading for a 300Gb local dataset shard size 1.1Gb on local HDD (40Mb/s read speed). This corresponds almost exactly to total data divided by reading speed implying that it reads the entire dataset at each load.

Stack details:
=========

> GCC version: Could not collect
> Clang version: Could not collect
> CMake version: Could not collect
> 
> Python version: 3.7 (64-bit runtime)
> Is CUDA available: True
> CUDA runtime version: 10.2.89
> GPU models and configuration: GPU 0: GeForce GTX 1050
> Nvidia driver version: 457.63
> cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\cudnn64_7.dll
> HIP runtime version: N/A
> MIOpen runtime version: N/A
> 
> Versions of relevant libraries:
> [pip3] datasets==1.6.2
> [pip3] transformers==4.5.1
> [pip3] numpy==1.19.1
> [pip3] numpydoc==1.1.0
> [pip3] pytorch-metric-learning==0.9.98
> [pip3] torch==1.8.1
> [pip3] torchaudio==0.8.1
> [pip3] torchvision==0.2.2
> [conda] blas                      2.16                        mkl    conda-forge
> [conda] cudatoolkit               10.2.89              hb195166_8    conda-forge
> [conda] libblas                   3.8.0                    16_mkl    conda-forge
> [conda] libcblas                  3.8.0                    16_mkl    conda-forge
> [conda] liblapack                 3.8.0                    16_mkl    conda-forge
> [conda] liblapacke                3.8.0                    16_mkl    conda-forge
> [conda] mkl                       2020.1                      216
> [conda] numpy                     1.19.1           py37hae9e721_0    conda-forge
> [conda] numpydoc                  1.1.0                      py_1    conda-forge
> [conda] pytorch                   1.8.1           py3.7_cuda10.2_cudnn7_0    pytorch
> [conda] pytorch-metric-learning   0.9.98             pyh39e3cac_0    metric-learning
> [conda] torchaudio                0.8.1                      py37    pytorch
> [conda] torchvision               0.2.2                      py_3    pytorch
Hi,

I reported too slow data fetching when data is large(#2210) a couple of weeks ago, and @lhoestq referred me to the fix (#2122).
However, the problem seems to persist. Here is the profiled results:


1) Running with 60GB
```
Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  517.96         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------
model_backward                     	|  0.26144        	|100            	|  26.144         	|  5.0475         	|
model_forward                      	|  0.11123        	|100            	|  11.123         	|  2.1474         	|
get_train_batch                    	|  0.097121       	|100            	|  9.7121         	|  1.8751         	|
```


3) Running with 600GB, datasets==1.6.0
```
Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  4563.2         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------
get_train_batch                    	|  5.1279         	|100            	|  512.79         	|  11.237         	|
model_backward                     	|  4.8394         	|100            	|  483.94         	|  10.605         	|
model_forward                      	|  0.12162        	|100            	|  12.162         	|  0.26652        	|
```

I see that `get_train_batch` lags when data is large. Could this be related to different issues?
I would be happy to provide necessary information to investigate.
Slow dataloading with big datasets issue persists 
 Hi,

I reported too slow data fetching when data is large(#2210) a couple of weeks ago, and @lhoestq referred me to the fix (#2122).
However, the problem seems to persist. Here is the profiled results:


1) Running with 60GB
```
Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  517.96         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------
model_backward                     	|  0.26144        	|100            	|  26.144         	|  5.0475         	|
model_forward                      	|  0.11123        	|100            	|  11.123         	|  2.1474         	|
get_train_batch                    	|  0.097121       	|100            	|  9.7121         	|  1.8751         	|
```


3) Running with 600GB, datasets==1.6.0
```
Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  4563.2         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------
get_train_batch                    	|  5.1279         	|100            	|  512.79         	|  11.237         	|
model_backward                     	|  4.8394         	|100            	|  483.94         	|  10.605         	|
model_forward                      	|  0.12162        	|100            	|  12.162         	|  0.26652        	|
```

I see that `get_train_batch` lags when data is large. Could this be related to different issues?
I would be happy to provide necessary information to investigate. 
 @lhoestq and @hwijeen

Despite upgrading to datasets 1.6.2, still experiencing extremely slow (2h00) loading for a 300Gb local dataset shard size 1.1Gb on local HDD (40Mb/s read speed). This corresponds almost exactly to total data divided by reading speed implying that it reads the entire dataset at each load.

Stack details:
=========

> GCC version: Could not collect
> Clang version: Could not collect
> CMake version: Could not collect
> 
> Python version: 3.7 (64-bit runtime)
> Is CUDA available: True
> CUDA runtime version: 10.2.89
> GPU models and configuration: GPU 0: GeForce GTX 1050
> Nvidia driver version: 457.63
> cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\cudnn64_7.dll
> HIP runtime version: N/A
> MIOpen runtime version: N/A
> 
> Versions of relevant libraries:
> [pip3] datasets==1.6.2
> [pip3] transformers==4.5.1
> [pip3] numpy==1.19.1
> [pip3] numpydoc==1.1.0
> [pip3] pytorch-metric-learning==0.9.98
> [pip3] torch==1.8.1
> [pip3] torchaudio==0.8.1
> [pip3] torchvision==0.2.2
> [conda] blas                      2.16                        mkl    conda-forge
> [conda] cudatoolkit               10.2.89              hb195166_8    conda-forge
> [conda] libblas                   3.8.0                    16_mkl    conda-forge
> [conda] libcblas                  3.8.0                    16_mkl    conda-forge
> [conda] liblapack                 3.8.0                    16_mkl    conda-forge
> [conda] liblapacke                3.8.0                    16_mkl    conda-forge
> [conda] mkl                       2020.1                      216
> [conda] numpy                     1.19.1           py37hae9e721_0    conda-forge
> [conda] numpydoc                  1.1.0                      py_1    conda-forge
> [conda] pytorch                   1.8.1           py3.7_cuda10.2_cudnn7_0    pytorch
> [conda] pytorch-metric-learning   0.9.98             pyh39e3cac_0    metric-learning
> [conda] torchaudio                0.8.1                      py37    pytorch
> [conda] torchvision               0.2.2                      py_3    pytorch