The choice of parameters, and the design of the network architecture are important factors affecting the performance of deep neural networks. Genetic Algorithms (GA) have been used before to determine parameters of a network. Yet, GAs perform a finite search over a discrete set of pre-defined candidates, and cannot, in general, generate unseen configurations. In this paper, to move from exploration to exploitation, we propose a novel and systematic method that autonomously and simultaneously optimizes multiple parameters of any deep neural network by using a GA aided by a bi-generative adversarial network (Bi-GAN). The proposed Bi-GAN allows the autonomous exploitation and choice of the number of neurons, for fully-connected layers, and number of filters, for convolutional layers, from a large range of values. Our proposed Bi-GAN involves two generators, and two different models compete and improve each other progressively with a GAN-based strategy to optimize the networks during GA evolution. Our proposed approach can be used to autonomously refine the number of convolutional layers and dense layers, number and size of kernels, and the number of neurons for the dense layers; choose the type of the activation function; and decide whether to use dropout and batch normalization or not, to improve the accuracy of different deep neural network architectures. Without loss of generality, the proposed method has been tested with the ModelNet database, and compared with the 3D Shapenets and two GA-only methods. The results show that the presented approach can simultaneously and successfully optimize multiple neural network parameters, and achieve higher accuracy even with shallower networks.
The genetic algorithm-based optimization uses a given set of blueprints and models, i.e. it performs a finite search over a discrete set of candidates. Thus, GAs, in general, cannot generate unseen configurations, and they can only make a combination of preset parameters. GAs are good at searching better solutions from limited possibilities, such as type of layers and activation functions. However, it cannot search for a solution which is not defined before. In addition, the complexity of GAs increases significantly when the number of choices increases to large scale. Rylander @cite_3 has shown that the generations needed for convergence increases exponentially with the node size.
@cite_3
130397803
A genetic algorithm is a biologically inspired method for function optimization that is loosely based on the theory of evolution. It is typically used when there is little knowledge of the solution space or when the search space is prohibitively large. Despite years of successful application to a variety of problems ranging from superstructure design to simulation of bacteria, little has been done to characterize the complexity of problems as they relate to this method. Complexity is critical however. Though the metaphor is seductive, the genetic algorithm is only a simulation of evolution. The accuracy of this simulation changes daily as more knowledge is gained from the biological communities and as computer researchers find better methods for solving problems regardless of the strict adherence to the biological inspiration. In the end, the genetic algorithm is a search method that must be implemented as a program consisting of loops and if statements. Given this, it seems reasonable to evaluate whether programs developed using this method provide any advantage in terms of efficiency over other methods. To do this, there must first be a way for evaluating the complexity of problems specifically for the genetic algorithm. This technique must enable a direct comparison between the GA-complexity of problems and what is already known about the complexity of problems. In this way it is possible to evaluate whether there is a “gain” in using genetic algorithms beyond programmer ergonomics. This dissertation describes a method for evaluating the complexity of a problem specifically for genetic algorithms. The method is used to define two specific genetic algorithm complexity classes. GA-hardness is defined as well as a method for GA reduction. In addition, the complexity of problems specifically for Genetic Programming (GP) is analyzed. Finally, the impact of quantum computers upon the complexity classes for evolutionary computation is examined. All of these areas are presented as peer-reviewed publications that have been developed as part of this research.