In deep learning, performance is strongly affected by the choice of architecture
 and hyperparameters. While there has been extensive work on automatic hyperpa-
 rameter optimization for simple spaces, complex spaces such as the space of deep
 architectures remain largely unexplored. As a result, the choice of architecture is
 done manually by the human expert through a slow trial and error process guided
 mainly by intuition. In this paper we describe a framework for automatically
 designing and training deep models. We propose an extensible and modular lan-
 guage that allows the human expert to compactly represent complex search spaces
 over architectures and their hyperparameters. The resulting search spaces are tree-
 structured and therefore easy to traverse. Models can be automatically compiled to
 computational graphs once values for all hyperparameters have been chosen. We
 can leverage the structure of the search space to introduce different model search
 algorithms, such as random search, Monte Carlo tree search (MCTS), and sequen-
 tial model-based optimization (SMBO). We present experiments comparing the
 different algorithms on CIFAR-10 and show that MCTS and SMBO outperform
 random search. We also present experiments on MNIST, showing that the same
 search space achieves near state-of-the-art performance with a few samples. These
 experiments show that our framework can be used effectively for model discov-
 ery, as it is possible to describe expressive search spaces and discover competitive
 models without much effort from the human expert. Code for our framework and
 experiments has been made publicly available Deep learning has seen a surge in popularity due to breakthroughs in applications such as computer vision, natural language processing, and reinforcement learning BID12 Karpathy & FeiFei, 2015; BID24 ). An important observation in much of the recent work is that complex architectures are important for achieving high performance BID12 BID20 . Larger datasets and more powerful computing infrastructures are likely to increase our ability to effectively train larger, deeper, and more complex architectures. However, improving the performance of a neural network is not as simple as adding more layers or parameters-it often requires clever ideas such as creating more branches or adding skip connections BID12 . Even popular techniques such as dropout BID27 and batch normalization BID14 do not always lead to better performance, and need to be judiciously applied to be helpful.Currently, choosing appropriate values for these architectural hyperparameters requires close supervision by a human expert, in a trial and error manual search process largely guided by intuition. The expert is burdened by having to make the large number of choices involved in the specification of a deep model. Choices interact in non-obvious ways and strongly impact performance. The typical workflow has the expert specify a single model, train it, and compute a validation score. Based on the validation score, previous experience, and information gathered during training, the expert decides if the trained model is satisfactory or not. If the model is considered unsatisfactory, the expert has to think about model variations that may lead to better performance.From the perspective of the expert, it would be convenient to search over architectures automatically, just as we search over simple scalar hyperparameters, such as the learning rate and the regularization coefficient. Ideally, the expert would have control in setting up the search space to incorporate inductive biases about the task being solved and constraints about computational resources. Prior to this work, achieving this goal was hard because expressing model search spaces using general hyperparameter optimization tools requires the human expert to manually distill a set of relevant scalar architectural hyperparameters.The main contributions of our work are 1. a modular, compositional, and extensible language for compactly representing expressive search spaces over models that (a) gives control to the human expert over what model variations to consider; (b) makes it easy to automatically search for performant models in the search space; (c) allows models to be directly compiled to computational graphs without the human expert having to write additional code. 2. model search algorithms that rely on the tree-structured search spaces induced by our language to systematically and efficiently search for performant models; namely, we (a) show that by using constructs in our language, even random search can be effective; (b) compare different model search algorithms experimentally, and show that random search is outperformed by algorithms that leverage the structure of the search space to generalize more effectively across different models.The main differences between our work and previous work are that we develop a modular, composable and extensible language, focusing on the problem of searching over deep architectures. This focus allows the expert to compactly set up a search space, search over it, and automatically compile models to their corresponding computational graphs. Our language can be seen as an effort to combine the functionalities of a deep model specification language (e.g., Tensorflow BID0 ) and a structured hyperparameter search language (e.g., Hyperopt BID32 ). We described a framework for automatically designing and training deep models. This framework consists of three fundamental components: the model search space specification language, the model search algorithm, and the model evaluation algorithm. The model search space specification language is composable, modular, and extensible, and allows us to easily define expressive search spaces over architectures. The model evaluation algorithm determines how to compute a score for a model in the search space. Models can be automatically compiled to their corresponding computational graphs. Using the model search space specification language and the model evaluation algorithm, we can introduce model search algorithms for exploring the search space. Using our framework, it is possible to do random search over interesting spaces of architectures without much effort from the expert. We also described more complex model search algorithms, such as MCTS, MCTS with tree restructuring, and SMBO. We present experiments on CIFAR-10 comparing different model search algorithms and show that MCTS with tree restructuring and SMBO outperform random search. Code for our framework and experiments has been made publicly available. We hope that this paper will lead to more work and better tools for automatic architecture search.
We describe a modular and composable language for describing expressive search spaces over architectures and simple model search algorithms applied to these search spaces. 
{"src": "scitldr-AIC_train", "paper_id": "rkTBjG-AZ"}