Blockwise knowledge distillation (BKD) is done by training every block for a single epoch. During this epoch, we apply a cosine learning rate schedule [1]} considering 20 steps, an initial learning rate of 0.01, a batch size of 256, the Adam [2]} optimizer, and random cropping and flipping as data augmentation.
