Is dataset iterative or not?
Hi Huggingface/Datasets team,
I want to use the datasets inside Seq2SeqDataset here
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py
and there I need to return back each line from the datasets and I am not
sure how to access each line and implement this?
It seems it also has get_item attribute? so I was not sure if this is
iterative dataset? or if this is non-iterable datasets?
thanks.



On Mon, Nov 9, 2020 at 10:18 AM Quentin Lhoest <notifications@github.com>
wrote:

> Hello !
> Could you give more details ?
>
> If you mean iter through one dataset then yes, Dataset object does
> implement the __iter__ method so you can use
>
> for example in dataset:
>     # do something
>
> If you want to iter through several datasets you can first concatenate them
>
> from datasets import concatenate_datasets
> new_dataset = concatenate_datasets([dataset1, dataset2])
>
> Let me know if this helps !
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/815#issuecomment-723881199>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYRLSSYW6NZN2HYDBTSO6XV5ANCNFSM4TPB7OWA>
> .
>

Hi
I want to use your library for large-scale training, I am not sure if this is implemented as iterative datasets or not?
could you provide me with example how I can use datasets as iterative datasets?
thanks
Is dataset iterative or not? 
 Hi
I want to use your library for large-scale training, I am not sure if this is implemented as iterative datasets or not?
could you provide me with example how I can use datasets as iterative datasets?
thanks 
 Hi Huggingface/Datasets team,
I want to use the datasets inside Seq2SeqDataset here
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py
and there I need to return back each line from the datasets and I am not
sure how to access each line and implement this?
It seems it also has get_item attribute? so I was not sure if this is
iterative dataset? or if this is non-iterable datasets?
thanks.



On Mon, Nov 9, 2020 at 10:18 AM Quentin Lhoest <notifications@github.com>
wrote:

> Hello !
> Could you give more details ?
>
> If you mean iter through one dataset then yes, Dataset object does
> implement the __iter__ method so you can use
>
> for example in dataset:
>     # do something
>
> If you want to iter through several datasets you can first concatenate them
>
> from datasets import concatenate_datasets
> new_dataset = concatenate_datasets([dataset1, dataset2])
>
> Let me know if this helps !
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/815#issuecomment-723881199>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYRLSSYW6NZN2HYDBTSO6XV5ANCNFSM4TPB7OWA>
> .
>
