A key challenge in leveraging data augmentation for neural network training is choosing an effective augmentation policy from a large search space of candidate operations. Properly chosen augmentation policies can lead to significant generalization improvements; however, state-of-the-art approaches such as AutoAugment are computationally infeasible to run for the ordinary user. In this paper, we introduce a new data augmentation algorithm, Population Based Augmentation (PBA), which generates nonstationary augmentation policy schedules instead of a fixed augmentation policy. We show that PBA can match the performance of AutoAugment on CIFAR-10, CIFAR-100, and SVHN, with three orders of magnitude less overall compute. On CIFAR-10 we achieve a mean test error of 1.46 , which is a slight improvement upon the current state-of-the-art. The code for PBA is open source and is available at this https URL.
We develop parallel predictive entropy search (PPES), a novel algorithm for Bayesian optimization of expensive black-box objective functions. At each iteration, PPES aims to select a batch of points which will maximize the information gain about the global maximizer of the objective. Well known strategies exist for suggesting a single evaluation point based on previous observations, while far fewer are known for selecting batches of points to evaluate in parallel. The few batch selection schemes that have been studied all resort to greedy methods to compute an optimal batch. To the best of our knowledge, PPES is the first non-greedy batch Bayesian optimization strategy. We demonstrate the benefit of this approach in optimization performance on both synthetic and real world applications, including problems in machine learning, rocket science and robotics.    An apparatus and method are provided for testing memory circuits in a microprocessor. The apparatus includes test management logic and test execution logic located within the microprocessor. The test management logic has a non-specific test program stored therein, and it accepts test parameters provided by an external test controller. The test parameters are applied to the non-specific test program to produce a specific test program. The test execution logic executes the specific test program to test the memory circuits within the microprocessor at the internal speed of the microprocessor.  Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisation algorithm. In this work we present , a simple asynchronous optimisation algorithm which effectively utilises a fixed computational budget to jointly optimise a population of models and their hyperparameters to maximise performance. Importantly, PBT discovers a schedule of hyperparameter settings rather than following the generally sub-optimal strategy of trying to find a single fixed set to use for the whole course of training. With just a small modification to a typical distributed hyperparameter training framework, our method allows robust and reliable training of models. We demonstrate the effectiveness of PBT on deep reinforcement learning problems, showing faster wall-clock convergence and higher final performance of agents by optimising over a suite of hyperparameters. In addition, we show the same method can be applied to supervised learning for machine translation, where PBT is used to maximise the BLEU score directly, and also to training of Generative Adversarial Networks to maximise the Inception score of generated images. In all cases PBT results in the automatic discovery of hyperparameter schedules and model selection which results in stable training and better final performance. Mutations are required for adaptation, yet most mutations with phenotypic effects are deleterious. As a consequence, the mutation rate that maximizes adaptation will be some intermediate value. This abstract summarizes a previous publication in which we used Avida, a well-studied artificial life platform, to investigate the ability of natural selection to adjust and optimize mutation rates. Our initial experiments occurred in a previously studied environment with a complex fitness landscape ( Nature, 423, 2003) where Avidians were rewarded for performing any of nine logic tasks. We assessed the optimal mutation rate by empirically determining which unchanging mutation rate produced the highest rate of adaptation. Then, we allowed mutation rates to evolve and we evaluated their proximity to the optimum. Although we chose conditions favorable for mutation rate optimization (asexual organisms not yet adapted to a new environment), the evolved rates were invariably far below the optimum across a wide range of experimental parameter settings (Fig. 1). We hypothesized that the reason mutation rates evolved to be suboptimal was the ruggedness of fitness landscapes. To test this hypothesis, we created a simplified 'counting ones' landscape without any fitness valleys and found that, in such conditions, populations evolved near-optimal mutation rates (Fig. 2, top row). In contrast, once moderate fitness valleys were added to this simple landscape, the ability of evolving populations to find the optimal mutation rate was lost (Fig. 2, bottom two rows). Additional experiments revealed that lowering the rate at which mutation rates evolved did not preclude the evolution of suboptimal mutation rates (see original manuscript). We conclude that rugged fitness landscapes can prevent the evolution of mutation rates that are optimal for long-term adaptation because of the short-term costs of traversing fitness valleys. This finding has important implications for evolutionary research in both biological and computational realms.    Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While current methods offer efficiencies by adaptively choosing new configurations to train, an alternative strategy is to adaptively allocate resources across the selected configurations. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinitely many armed bandit problem where allocation of additional resources to an arm corresponds to training a configuration on larger subsets of the data. We introduce Hyperband for this framework and analyze its theoretical properties, providing several desirable guarantees. We compare Hyperband with state-of-the-art Bayesian optimization methods and a random search baseline on a comprehensive benchmark including 117 datasets. Our results on this benchmark demonstrate that while Bayesian optimization methods do not outperform random search trained for twice as long, Hyperband in favorable settings offers valuable speedups.