Transformer Class on dataset
Ok thanks.

But, Dataset has various flavors.
In current design of Dataset,
   how the serialization on disk is done (?)


The main issue is serialization 
of   newdataset= Transform(Dataset)
 (ie thats why am referring to Out Of memory dataset…):

   Should be part of Transform or part of dataset ?




Maybe, not, since the output is aimed to feed model in memory (?)








> On Jul 7, 2021, at 18:04, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> We can be pretty flexible and not impose any constraints for transforms.
> 
> Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like map work in a batched fashion to not fill up your RAM. So this shouldn't be an issue
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).






Transformer Class on dataset 
 Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).





 
 Ok thanks.

But, Dataset has various flavors.
In current design of Dataset,
   how the serialization on disk is done (?)


The main issue is serialization 
of   newdataset= Transform(Dataset)
 (ie thats why am referring to Out Of memory dataset…):

   Should be part of Transform or part of dataset ?




Maybe, not, since the output is aimed to feed model in memory (?)








> On Jul 7, 2021, at 18:04, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> We can be pretty flexible and not impose any constraints for transforms.
> 
> Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like map work in a batched fashion to not fill up your RAM. So this shouldn't be an issue
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
