In knowledge distillation a large cumbersome CNN called teacher network is used to train a smaller network referred to as student network [1]}. The student network is designed by manually removing structured elements such as filters, layers, blocks from an existing network in an adhoc manner. Despite several successes in training a student network [2]}, [3]}, [4]}, [4]}, designing a student network is still an open problem and is usually performed via hit and try which is a tedious process.
