Several recent studies consider hardware performance for NAS. Authors in [1]} introduce a growing and pruning strategy that automatically maximizes the test accuracy and minimizes the FLOPs of neural architectures during training.
A platform-aware NAS targeting mobile devices is proposed in [2]};
the objective is to maximize the model accuracy with an upper bound on latency. Authors in [3]} create a latency-aware loss function to perform differentiable NAS. The latency of DNNs is estimated through a lookup table which consists of the latency of each operation/layer.
However, both of these studies consider latency as the only metric for hardware performance. Authors in [4]} propose a hardware-aware NAS framework to design convolutional neural networks. Specifically, by building analytical latency, power, and memory models, they create a hardware-aware optimization methodology to search for the optimal architecture that meets the hardware budgets.
Authors in [5]} consider latency, energy, and area as metrics for hardware performance while performing NAS.
Also, a reinforcement learning (RL)-based controller is adopted to tune the network architecture and device parameters.
The resulting network is retrained to evaluate the model accuracy.
There are two major drawbacks of this approach.
First, RL is a slow-converging process that prohibits fast exploration of the design space.
Second, retraining the network further exacerbates the search time leading to hundreds of GPU hours needed for real applications [6]}.
Furthermore, most existing hardware-aware NAS approaches explicitly optimize the architectures for a specific hardware platform [7]}, [3]}, [9]}. Hence, if we switch to some new hardware, we need to repeat the entire NAS process, which is very time-consuming under the existing NAS frameworks [7]}, [3]}, [9]}. The demand for reducing the overhead of adaptation to new hardware motivates us to improve the transferability of hardware-aware NAS methodology.
