Architecture engineering is concerned with determining the most appropriate operations for the network, their hyper-parameters (e.g. the number of neurons for fully connected layers, or the number of filters or kernel size for convolutional layers), and the connectivity of all the operations. Generally, practitioners propose novel operations to validate various architectures and tasks in order to improve performance on specific tasks. As a result, developing a novel operation remains a time-consuming and costly process. It necessitates a manual search for the optimal configuration. As a result, it is prone to failure when practitioners lack computational resources. An alternative approach focus on automatically finding the network architecture design using Neural architecture search (NAS methods) [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]} in lieu of a manual design. Given a set of data and a performance metric for a learning task, a NAS algorithm attempts to find the optimal architecture concerning a search strategy. It can be viewed as an optimization problem in the space of an architecture network defined by a collection of operations and their possible combinations. Recently, it was demonstrated empirically on several applications that architectures discovered using NAS outperform those discovered manually.
