We assume there are \(M\)  learners in total. For each learner \(m\) , it has a training set \(D^{(\textrm {tr})}_m\)  and optionally a validation set \(D^{(\textrm {val})}_m\) . Meanwhile, all learners share a common collection of auxiliary datasets \(\mathcal {F}\) , which could be unlabeled datasets used for self-supervised pretraining [1]}, additional labeled datasets used for validation, and so on. The learner \(m\)  has one or more sets of learnable parameters \(\lbrace W_i^{(m)}\rbrace _{i=1}^{N_m}\) . The learnable parameters could be network weights, architectures, hyperparameters, weights of training examples, etc.
