Deep neural networks lie at the heart of many of the artificial intelligence applications that are ubiquitous in our society. Over the past several years, methods for training these networks have become more automatic [1]}, [2]}, [3]}, [4]}, [5]} but still remain more an art than a science. This paper introduces the high-level concept of general cyclical training as another step in making it easier to optimally train neural networks. We argue that many of the settings that are held constant throughout training need not be and training improves when they are not constant.
