sample multiple datasets 
Hi
I am dealing with multiple datasets, I need to have a dataloader over them with a condition that in each batch data samples are coming from one of the datasets. My main question is: 
-  I need to have a way to sample the datasets first with some weights, lets say 2x dataset1 1x dataset2, could you point me how I can do it

sub-questions:
- I want to concat sampled datasets and define one dataloader on it, then I need a way to make sure batches come from 1 dataset in each iteration, could you assist me how I can do?
- I use iterative-type of datasets, but I need a method of shuffling still since it brings accuracy performance issues if not doing it, thanks for the help. 
here I share my dataloader currently for multiple tasks: https://gist.github.com/rabeehkarimimahabadi/39f9444a4fb6f53dcc4fca5d73bf8195 

I need to train my model distributedly with this dataloader, "MultiTasksataloader", currently this does not work in distributed fasion,
to save on memory I tried to use iterative datasets, could you have a look in this dataloader and tell me if this is indeed the case? not sure how to make datasets being iterative to not load them in memory, then I remove the sampler for dataloader, and shard the data per core, could you tell me please how I should implement this case in datasets library? and how do you find my implementation in terms of correctness? thanks 
 Hi @rabeehkarimimahabadi any luck with updating the multi-task data loader to work with distributed training? Hi @pushkalkatara yes I solved it back then, here please find my implementation https://github.com/rabeehk/hyperformer/blob/main/hyperformer/data/multitask_sampler.py  Thanks @rabeehk for sharing. 

The sampler basically returns a list of integers to sample from each task's dataset. I was wondering how to use it with two `torch.Dataset` of different tasks. Also, do I need to shard across processes while creating an Iterable Dataset?
 We now have `interleave_datasets` in the API that allows you to cycle/sample with probabilities (with various stopping strategies) through a list of datasets. However, more specific behavior should be implemented manually. Hi @mariosasko, @pushkalkatara 

I have multi dataset for ASR task. I have multilingual such as: English, China, Japan, Vietnamese. Each dataset is loaded as dataset. Now I training, I want to equal sampler, it means each batch loader in training has equal number sampler in each language.

For example: Batch_size = 16:
English sample:4
China sample:4
Japan sample:4
Vietnamese sample:4 

How can I do it? I saw that  interleave_datasets only splits data with probability an concatenates all of them after that, it not affect in training data loader. Forgiving me if I wrong.

Thank you for your help.