During architecture search, we perform two rounds of learning, with an order of CIFAR-100, CIFAR-10, CIFAR-100, CIFAR-10. The tradeoff parameter \(\beta \)  in interleaving learning was set to 100. The network of the feature extractor is a stack of 8 cells. Each cell has 7 nodes. We set the initial channel number to 16. We optimized the architecture variables using Adam [1]}. The learning rate was set to 3e-5 for IL-DARTS, 6e-4 for IL-P-DARTS, and 3e-3 for IL-PC-DARTS. The weight decay was set to 1e-3.
We optimized the network weights using SGD. The initial learning rate was set to 0.025 for IL-DARTS and IL-P-DARTS and 0.1 for IL-PC-DARTS. The batch size was set to 64 for IL-DARTS and IL-P-DARTS and 256 for IL-PC-DARTS. The epoch number was set to 50 for IL-DARTS and IL-PC-DARTS and 25 for IL-P-DARTS. Weight decay was set to 3e-4 and momentum was set to 0.9. Cosine decay scheduler was used for scheduling the learning rate. The search was performed on a single Teslav100 GPU.
