Similarly, Blockwisely-Supervised NAS (DNA) [1]}, splits NAS into two phases: the creation of a ranking model for a search space and a targeted search to find the highest-ranked models at a given constraint. To build this ranking model, DNA uses blockwise knowledge distillation (BKD) to build a relative ranking of all possible networks in a given search space. The best networks are then trained and verified. It is crucial to note that it is BKD that
enables the diverse search for optimal attention mechanisms, activation functions, and channel scaling. However, DNA has three disadvantages: (1) the ranking model fails when ranking large and diverse search spaces (Section REF ), (2) the ranking only holds within a search space and does not allow the comparison of different spaces, and (3) because of the reliance on training subsampled architectures from scratch, the method is not competitive in terms of search time. This work, DONNA, addresses all these issues. In summary, DONNA differs from prior work on these key aspects:
