In this section, we present the details of the interleaving learning framework. There are \(K\)  learners. Each learner learns to perform a task. These tasks could be the same, e.g., image classification on CIFAR-10; or different, e.g., image classification on CIFAR-10, image classification on ImageNet [1]}, object detection on MS-COCO [2]}, etc. Each learner \(k\)  has a training dataset \(D_k^{(\textrm {tr})}\)  and a validation dataset \(D_k^{(\textrm {val})}\) .
Each learner has a data encoder and a task-specific head performing the target task. For example, if the task is image classification, the data encoder could be a convolutional neural network extracting visual features of the input images and the task-specific head could be a multi-layer perceptron which takes the visual features of an image extracted by the data encoder as input and predicts the class label of this image. We assume the architecture of the data encoder in each learner is learnable. The data encoders of all learners share the same architecture, but their weight parameters could be different in different learners. The architectures of task-specific heads are manually designed by humans and they could be different in different learners. The \(K\)  learners perform \(M\)  rounds of interleaving learning with the following order:
\(\underbrace{l_1,l_2,\cdots ,l_K}_{\textrm {Round }1}\underbrace{l_1,l_2,\cdots ,l_K}_{\textrm {Round }2}\cdots \underbrace{l_1,l_2,\cdots ,l_K}_{\textrm {Round }m}\cdots \underbrace{l_1,l_2,\cdots ,l_K}_{\textrm {Round }M}\) 
