Transformer Class on dataset
Thanks for reply.

What would be the constraints
to have
Dataset —> Dataset consistency ?

Main issue would be
larger than memory dataset and
serialization on disk.

Technically,
one still process at atomic level
and try to wrap the full results
into Dataset…. (!)

What would you think ?








> On Jul 7, 2021, at 16:51, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.
> 
> I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).






Transformer Class on dataset 
 Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).





 
 Thanks for reply.

What would be the constraints
to have
Dataset —> Dataset consistency ?

Main issue would be
larger than memory dataset and
serialization on disk.

Technically,
one still process at atomic level
and try to wrap the full results
into Dataset…. (!)

What would you think ?








> On Jul 7, 2021, at 16:51, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.
> 
> I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
