Architecture search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: (1) the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption; (2)most architecture search methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the Pareto-front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates children networks that are warmstarted with the predictive performance of their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions allows finding models that are on par or even outperform different-sized NASNets, MobileNets, MobileNets V2 and Wide Residual Networks on CIFAR-10 and ImageNet64x64 within only one week on eight GPUs, which is about 20-40x less compute power than previous architecture search methods that yield state-of-the-art performance. Deep learning has enabled remarkable progress on a variety of perceptual tasks, such as image recognition BID12 , speech recognition , and machine translation BID0 . One crucial aspect for this progress are novel neural architectures BID25 He et al., 2016; BID7 . Currently employed architectures have mostly been developed manually by human experts, which is a time-consuming and error-prone process. Because of this, there is growing interest in automatic architecture search methods (Elsken et al., 2018) . Some of the architectures found in an automated way have already outperformed the best manually-designed ones; however, algorithms such as by BID32 ; ; BID20 BID36 for finding these architectures require enormous computational resources often in the range of thousands of GPU days.Prior work on architecture search has typically framed the problem as a single-objective optimization problem. However, most applications of deep learning do not only require high predictive performance on unseen data but also low resource-consumption in terms of, e.g., inference time, model size or energy consumption. Moreover, there is typically an implicit trade-off between predictive performance and consumption of resources. Recently, several architectures have been manually designed that aim at reducing resource-consumption while retaining high predictive performance BID8 BID22 . Automatically found neural architectures have also been down-scaled to reduce resource consumption . However, very little previous work has taken the trade-off between resource-consumption and predictive performance into account during automatic architecture search.In this work, we make the following two main contributions:1. To overcome the need for thousands of GPU days BID32 BID21 , we make use of operators acting on the space of neural network architectures that preserve the function a network represents, dubbed network morphisms (Chen et al., 2015; BID27 , obviating training from scratch and thereby substantially reducing the required training time per network. This mechanism can be interpreted as Lamarckian inheritance in the context of evolutionary algorithms, where Lamarckism refers to a mechanism which allows passing skills acquired during an individual's lifetime (e.g., by means of learning), on to children by means of inheritance. Since network morphisms are limited to solely increasing a network's size (and therefore likely also resource consumption), we introduce approximate network morphisms (Section 3.2) to also allow shrinking networks, which is essential in the context of multi-objective search. The proposed Lamarckian inheritance mechanism could in principle be combined with any evolutionary algorithm for architecture search, or any other method using (a combination of) localized changes in architecture space.2. We propose a Lamarckian Evolutionary algorithm for Multi-Objective Neural Architecture DEsign, dubbed LEMONADE, Section 4, which is suited for the joint optimization of several objectives, such as predictive performance, inference time, or number of parameters. LEMONADE maintains a population of networks on an approximation of the Pareto front of the multiple objectives. In contrast to generic multi-objective algorithms, LEMONADE exploits that evaluating certain objectives (such as an architecture's number of parameters) is cheap while evaluating the predictive performance on validation data is expensive (since it requires training the model first). Thus, LEMONADE handles its various objectives differently: it first selects a subset of architectures, assigning higher probability to architectures that would fill gaps on the Pareto front for the "cheap" objectives; then, it trains and evaluates only this subset, further reducing the computational resource requirements during architecture search. In contrast to other multi-objective architecture search methods, LEMONADE (i) does not require to define a trade-off between performance and other objectives a-priori (e.g., by weighting objectives when using scalarization methods) but rather returns a set of architectures, which allows the user to select a suitable model a-posteriori; (ii) LEMONADE does not require to be initialized with well performing architectures; it can be initialized with trivial architectures and hence requires less prior knowledge. Also, LEMONADE can handle various search spaces, including complex topologies with multiple branches and skip connections.We evaluate LEMONADE for up to five objectives on two different search spaces for image classification: (i) non-modularized architectures and (ii) cells that are used as repeatable building blocks within an architecture BID31 and also allow transfer to other data sets. LEMONADE returns a population of CNNs covering architectures with 10 000 to 10 000 000 parameters.Within only 5 days on 16 GPUs, LEMONADE discovers architectures that are competitive in terms of predictive performance and resource consumption with hand-designed networks, such as MobileNet V2 BID22 , as well as architectures that were automatically designed using 40x greater resources and other multi-objective methods (Dong et al., 2018) . We have proposed LEMONADE, a multi-objective evolutionary algorithm for architecture search. The algorithm employs a Lamarckian inheritance mechanism based on (approximate) network morphism operators to speed up the training of novel architectures. Moreover, LEMONADE exploits the fact that evaluating several objectives, such as the performance of a neural network, is orders of magnitude more expensive than evaluating, e.g., a model's number of parameters. Experiments on CIFAR-10 and ImageNet64x64 show that LEMONADE is able to find competitive models and cells both in terms of accuracy and of resource efficiency.We believe that using more sophisticated concepts from the multi-objective evolutionary algorithms literature and using other network operators (e.g., crossovers and advanced compression methods) could further improve LEMONADE's performance in the future.
We propose a method for efficient Multi-Objective Neural Architecture Search based on Lamarckian inheritance and evolutionary algorithms.
{"src": "scitldr-AIC_train", "paper_id": "ByME42AqK7"}