Current NAS algorithms, however, suffer from several limitations. First, many optimization algorithms [1]}, [2]}, [3]}, [4]} target only a single deployment scenario: a hardware-agnostic complexity metric, a hardware platform, or different latency, energy, or accuracy requirements.
This means the search has to be repeated whenever any part of that scenario changes. Second, many methods cannot search in truly diverse search spaces, with different types of convolutional kernels, activation functions and attention mechanisms. Current methods either search through large and diverse spaces at a prohibitively expensive search cost [1]}, [2]}, or limit their applicability by trading search time for a more constrained and less diverse search [7]}, [3]}, [9]}, [10]}, [11]}, [12]}. Most of such speedups in NAS come from a reliance on weight sharing mechanisms, which require all architectures in the search space to be structurally similar. Thus, these works typically only search among micro-architectural choices such as kernel sizes, expansion rates, and block repeats and not among macro-architectural choices of layer types, attention mechanisms and activation functions. As such, they rely on prior expensive methods such as [1]}, [2]} for an optimal choice of macro-architecture.
