The principle of general cyclical training can extend to the network's architecture. Although it is currently unusual for the architecture to change while training, there exists work on growing a network during training [1]}, [2]}, [3]}. However, it is more practical to consider the the general cyclical training principle in the scenarios involving more than one network, such as with transfer learning/pre-training and with teacher-student/knowledge distillation [4]}, [5]}.
