To improve the scaling performance of NAS across different scenarios, it is critical to decouple the accuracy prediction of a model from the complexity objective. In Once-for-All (OFA) [1]} and [2]}, a large weight-sharing supernet is trained using progressive shrinking. This process allows the sampling of smaller subnets from the trained supernet that perform comparably with models that have been trained from scratch. A large number of networks can then be sampled to build an accuracy predictor for this search space, which in turn can be used in a scenario-aware evolutionary search, as in Figure REF (c).
Although similar to DONNA in this approach, OFA [1]} has several disadvantages. First, its search space's diversity is limited due to its reliance on progressive shrinking and weight sharing, which requires a fixed macro-architecture in terms of layer types, attention, activations, and channel widths.
Furthermore, progressive shrinking can only be parallelized in the batch dimension, limiting the maximum number of GPUs that can process in parallel. DONNA does not suffer from these constraints.
