Early approaches to NAS rely on reinforcement learning [1]}, [2]}, [3]} or evolutionary optimization [4]}. These methods allow for diverse search spaces, but at infeasibly high costs due to the requirement to train thousands of models for a number of epochs throughout the search. MNasNet [3]} for example uses up to 40,000 epochs in a single search. This process can be sped up by using weight sharing among different models, as in ENAS [6]}. However, this comes at the cost of a less diverse search space, as the subsampled models have to be similar for the weights to be shareable.
