We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour. Discovering high-performance neural network architectures required years of extensive research by human experts through trial and error. As far as the image classification task is concerned, state-ofthe-art convolutional neural networks are going beyond deep, chain-structured layout BID19 BID6 towards increasingly more complex, graph-structured topologies BID12 BID27 BID9 . The combinatorial explosion in the design space makes handcrafted architectures not only expensive to obtain, but also likely to be suboptimal in performance.Recently, there has been a surge of interest in using algorithms to automate the manual process of architecture design. Their goal can be described as finding the optimal architecture in a given search space such that the validation accuracy is maximized on the given task. Representative architecture search algorithms can be categorized as random with weights prediction BID1 , Monte Carlo Tree Search BID15 , evolution BID21 BID26 BID13 BID16 , and reinforcement learning BID0 BID30 BID31 BID29 , among which reinforcement learning approaches have demonstrated the strongest empirical performance so far.Architecture search can be computationally very intensive as each evaluation typically requires training a neural network. Therefore, it is common to restrict the search space to reduce complexity and increase efficiency of architecture search. Various constraints that have been used include: growing a convolutional "backbone" with skip connections BID16 , a linear sequence of filter banks BID1 , or a directed graph where every node has exactly two predecessors BID31 . In this work we constrain the search space by imposing a hierarchical network structure, while allowing flexible network topologies (directed acyclic graphs) at each level of the hierarchy. Starting from a small set of primitives such as convolutional and pooling operations at the bottom level of the hierarchy, higher-level computation graphs, or motifs, are formed by using lower-level motifs as their building blocks. The motifs at the top of the hierarchy are stacked multiple times to form the final neural network. This approach enables search algorithms to implement powerful hierarchical modules where any change in the motifs is propagated across the whole network immediately. This is analogous to the modularized design patterns used in many handcrafted architectures, e.g. VGGNet BID19 , ResNet BID6 , and Inception BID24 are all comprised of building blocks. In our case, a hierarchical architecture is discovered through evolutionary or random search.The evolution of neural architectures was studied as a sub-task of neuroevolution BID8 BID14 BID28 BID21 BID3 , where the topology of a neural network is simultaneously evolved along with its weights and hyperparameters. The benefits of indirect encoding schemes, such as multi-scale representations, have historically been discussed in BID5 ; BID11 BID20 BID22 . Despite these pioneer studies, evolutionary or random architecture search has not been investigated at larger scale on image classification benchmarks until recently BID16 BID13 BID26 BID1 BID15 . Our work shows that the power of simple search methods can be substantially enhanced using well-designed search spaces.Our experimental setup resembles BID31 , where an architecture found using reinforcement learning obtained the state-of-the-art performance on ImageNet. Our work reveals that random or evolutionary methods, which so far have been seen as less efficient, can scale and achieve competitive performance on this task if combined with a powerful architecture representation, whilst utilizing significantly less computational resources.To summarize, our main contributions are:1. We introduce hierarchical representations for describing neural network architectures. 2. We show that competitive architectures for image classification can be obtained even with simplistic random search, which demonstrates the importance of search space construction. 3. We present a scalable variant of evolutionary search which further improves the results and achieves the best published results 1 among evolutionary architecture search techniques. We have presented an efficient evolutionary method that identifies high-performing neural architectures based on a novel hierarchical representation scheme, where smaller operations are used as the building blocks to form the larger ones. Notably, we show that strong results can be obtained even using simplistic search algorithms, such as evolution or random search, when coupled with a well-designed architecture representation. Our best architecture yields the state-of-the-art result on A ARCHITECTURE VISUALIZATION Visualization of the learned cell and motifs of our best-performing hierarchical architecture. Note that only motifs 1,3,4,5 are used to construct the cell, among which motifs 3 and 5 are dominating.
In this paper we propose a hierarchical architecture representation in which doing random or evolutionary architecture search yields highly competitive results using fewer computational resources than the prior art.
{"src": "scitldr-AIC_train", "paper_id": "BJQRKzbA-"}