Starting from these overarching ideas, this paper proposes FLASH – a fast neural architecture search with hardware optimization – to address the drawbacks of current NAS techniques. FLASH delivers a neural architecture that is co-optimized with respect to accuracy and hardware performance. Specifically, by analyzing the topological property of neural architectures from a network science perspective, we propose a new topology-based metric, namely, the NN-Degree. We show that NN-Degree could indicate the test performance of a given architectures. This makes our proposed NAS training-free during the search process and accelerates NAS by orders of magnitude compared to state-of-the-art approaches. Then, we demonstrate that NN-Degree enables a lightweight accuracy predictor with only three parameters. Moreover, to improve the on-chip communication efficiency, we adopt the mesh-NoC for the IMC-based hardware. Based on the communication-optimized hardware architecture, we measure the hardware performance for a subset of neural networks from the NAS search space.
Then, we construct analytical models for the area, latency, and energy consumption of a neural network based on our optimized target hardware platform.
Unlike existing neural network-based and black-box style searching algorithms [1]}, the proposed NAS methodology enable searching across the entire search space via a mathematically rigorous and time-efficient optimization algorithm.
Consequently, our experimental evaluations show that FLASH significantly pushes forward the NAS frontier by enabling NAS in less than 0.1 seconds on a 20-core Intel Xeon CPU. Finally, we demonstrate that FLASH could be readily transferred to other hardware platforms (e.g., Raspberry Pi) only by fine-tuning the hardware performance models.
