We propose a genetic algorithm (GA) for hyperparameter optimization of artificial neural networks which includes chromosomal crossover as well as a decoupling of parameters (i.e., weights and biases) from hyperparameters (e.g., learning rate, weight decay, and dropout) during sexual reproduction. Children are produced from three parents; two contributing hyperparameters and one contributing the parameters. Our version of population-based training (PBT) combines traditional gradient-based approaches such as stochastic gradient descent (SGD) with our GA to optimize both parameters and hyperparameters across SGD epochs. Our improvements over traditional PBT provide an increased speed of adaptation and a greater ability to shed deleterious genes from the population. Our methods improve final accuracy as well as time to fixed accuracy on a wide range of deep neural network architectures including convolutional neural networks, recurrent neural networks, dense neural networks, and capsule networks.
This work first frames PBT as a genetic algorithm and then generalizes and extends it in Section . The original version of PBT can be thought of as a genetic algorithm with the following characteristics: overlapping generations, genes comprised of hyperparameters and parameters, GA generations running concurrently with SGD training epochs, asexual reproduction without crossover, two possibilities for viability selection (tournament and rank), and a fitness function customized for each of the NN objectives (e.g., using the BLUE score for neural machine translation @cite_22 ). Crossover (i.e., recombination), discussed in more detail in Section , combines hyperparameters from two parents to create a new child, as depicted in Figure . This provides not only a possibility for the aggregation of beneficial genes, but helps shed deleterious mutations from the population.
@cite_22
2101105183
Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations.