Transformer Class on dataset
Ok, sure.

Thanks for pointing on functional part.
My question is more
“Philosophical”/Design perspective.

There are 2 perspetive:
  Add transformation methods to 
     Dataset Class


 OR Create a Transformer Class
   which operates on Dataset Class.

T(Dataset) —> Dataset

datasetnew = MyTransform.transform(dataset)
datasetNew.save(path)


What would be the difficulty
of implementing a Transformer Class
operating at dataset level ?


thanks









> On Jul 6, 2021, at 22:00, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> There are already a few transformations that you can apply on a dataset using methods like dataset.map().
> You can find examples in the documentation here:
> https://huggingface.co/docs/datasets/processing.html
> 
> You can merge two datasets with concatenate_datasets() or do label extraction with dataset.map() for example
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).






Transformer Class on dataset 
 Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).





 
 Ok, sure.

Thanks for pointing on functional part.
My question is more
“Philosophical”/Design perspective.

There are 2 perspetive:
  Add transformation methods to 
     Dataset Class


 OR Create a Transformer Class
   which operates on Dataset Class.

T(Dataset) —> Dataset

datasetnew = MyTransform.transform(dataset)
datasetNew.save(path)


What would be the difficulty
of implementing a Transformer Class
operating at dataset level ?


thanks









> On Jul 6, 2021, at 22:00, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> There are already a few transformations that you can apply on a dataset using methods like dataset.map().
> You can find examples in the documentation here:
> https://huggingface.co/docs/datasets/processing.html
> 
> You can merge two datasets with concatenate_datasets() or do label extraction with dataset.map() for example
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
