In total, including the training of a reference model (450 epochs), \(450+1920+30\times 50=3870\)  epochs of training are required to build the accuracy predictor. This is less than \(10\times \)  the cost of training a single network from scratch to model the accuracy of more than 8 trillion architectures. Subsequently, any architecture can be selected and trained to full accuracy in 50 epochs, starting from the BKD initialization. Similarly, as further discussed in Appendix REF , an accuracy model for MobileNetV3 (1.2\(\times \) ) and EfficientNet-Compressed costs \(450+135+20\times 50=1585\)  epochs, roughly the same as training 4 models from scratch. Although this is a higher cost than OFA [1]}, it covers a much more diverse search space. OFA requires an equivalent, accounting for dynamic batch sizes [2]}, of \(180+125+2\times 150+4\times 150=1205\)  epochs of progressive shrinking with backpropagation on a large supernet. BKDNAS [3]} requires only \(450+16\times 20=770\)  epochs to build its ranking model, but 450 epochs to train models from scratch. Other methods like MnasNet [4]} can handle a similar diversity as DONNA, but typically require an order of magnitude longer search time (40000 epochs) for every deployment scenario. DONNA offers MNasNet-level diversity at a 2 orders of magnitude lower search cost. On top of that, BKD epochs are significantly faster than epochs on a full network, as BKD requires only partial computation of the reference model and backpropagation on a single block \(B_{n,m}\) . Moreover, and in contrast to OFA, all blocks \(B_{n,m}\)  can be trained in parallel since they are completely independent of each other. Table REF  quantifies the differences in search-time between these approaches.
