We present DONNA (Distilling Optimal Neural Network Architectures), a method that addresses both issues: it scales to multiple deployment scenarios with low additional cost, and performs rapid NAS in diverse search spaces. The method starts with a trained reference model. The first issue is resolved by splitting NAS into a scenario-agnostic training phase, and a scenario-aware search phase that requires only limited training, as depicted in Figure REF . After an accuracy predictor is built in the training phase, the search is executed quickly for each new deployment scenario, typically in the time-frame of hours, and only requiring minimal fine-tuning to finalize optimal models. Second, DONNA considers diverse macro-architectural choices in addition to micro-architectural choices, by creating this accuracy predictor through Blockwise Knowledge Distillation (BKD) [1]}, see Figure REF . This approach imposes little constraints on the macro- and micro-architectures under consideration, allowing a vast, diverse, and extensible search space. The DONNA pipeline yields state of the art network architectures, as illustrated for a Samsung S20 GPU in Figure REF (d). Finally, we use DONNA for rapid search space extension and exploration, and on-device model compression. This is possible as the DONNA accuracy predictor generalizes to architectures outside the original search space.
<FIGURE>