In another line of work, differentiable architecture search methods such as DARTS [1]}, FBNet [2]}, FBNetV2 [3]}, ProxylessNAS [4]}, AtomNAS [5]} and Single-Path NAS [6]} simultaneously optimize the weights of a large supernet and its architectural parameters.
This poses several impediments to scalable and scenario-aware NAS in diverse search spaces. First, in most of these works, different cell choices have to be available to the algorithm, ultimately limiting the space's size and diversity. While several works address this problem either by trading off the number of architecture parameters against the number of weights that are in GPU memory at a given time [7]}, by updating only a subset of the weights during the search [8]}, or by exploiting more granular forms of weight-sharing [6]}, the fundamental problem remains when new operations are introduced. Second, although differentiable search methods speed up a single search iteration, the search must be repeated for every scenario due to their coupling of accuracy and complexity. Differentiable methods also require differentiable cost models.
Typically these models use the sum of layer latencies as a proxy for the network latency, which can be inaccurate. This is especially the case in emerging depth-first processors [10]}, where intermediate results are stored in the local memory, making full-graph latency depend on layer sequences rather than on individual layers.
