Recently, there emerged revived interests of designing automatic programs (e.g., using genetic evolutionary algorithms) to optimize the structure of Convolutional Neural Networks (CNNs) for a specific task. The challenge in designing such programs lies in how to balance between large search space of the network structures and high computational costs. Existing works either impose strong restrictions on the search space or use enormous computing resources. In this paper, we study how to design a genetic programming approach for optimizing the structure of a CNN for a given task under limited computational resources yet without imposing strong restrictions on the search space. To reduce the computational costs, we propose two general strategies that are observed to be helpful: (i) aggressively selecting strongest individuals for survival and reproduction, and killing weaker individuals at a very early age; (ii) increasing mutation frequency to encourage diversity and faster evolution. The combined strategy with additional optimization techniques allows us to explore a large search space but with affordable computational costs. Our results on standard benchmark datasets (MNIST, SVHN, CIFAR-10, CIFAR-100) are competitive to similar approaches with significantly reduced computational costs.
Another fundamental issue in genetic evolutionary algorithm is the diversity of the population. A traditional approach for encouraging diversity of population is fitness sharing, where the fitness of each individual is scaled based on its proximity to others. It means that originally good solutions in densely populated regions will be given a lower fitness value than comparably good solutions in sparsely populated regions. All the three recent works @cite_16 @cite_4 @cite_28 did not use any type of fitness sharing to encourage diversity. @cite_4 , the authors simply use a very large population size (i.e., 1000) to increase the diversity. A key difference between our work and these previous work lies in the selection process and the number of mutation operations. The proposed solution takes both the limitations of computational resources and diversity of the population into account. As a result, even though we do not impose any strong restriction on the search space, we can use less computational costs to achieve competitive if not better prediction performance than @cite_16 @cite_28 @cite_4 .
@cite_28 @cite_16 @cite_4
2552732996 2071853901 2497618134 2138537392
Nowadays, in the world of limited attention, the techniques that maximize the spread of social influence are more than welcomed. Companies try to maximize their profits on sales by providing customers with free samples believing in the power of word-of-mouth marketing, governments and non-governmental organizations often want to introduce positive changes in the society by appropriately selecting individuals or election candidates want to spend least budget yet still win the election. In this work we propose the use of evolutionary algorithm as a mean for selecting seeds in social networks. By framing the problem as genetic algorithm challenge we show that it is possible to outperform well-known greedy algorithm in the problem of influence maximization for the linear threshold model in both: quality (up to 16 better) and efficiency (up to 35 times faster). We implemented these two algorithms by using GPGPU approach showing that also the evolutionary algorithm can benefit from GPU acceleration making it efficient and scaling better than the greedy algorithm. As the experiments conducted by using three real world datasets reveal, the evolutionary approach proposed in this paper outperforms the greedy algorithm in terms of the outcome and it also scales much better than the greedy algorithm when the network size is increasing. The only drawback in the GPGPU approach so far is the maximum size of the network that can be processed - it is limited by the memory of the GPU card. We believe that by showing the superiority of the evolutionary approach over the greedy algorithm, we will motivate the scientific community to look for an idea to overcome this limitation of the GPU approach - we also suggest one of the possible paths to explore. Since the proposed approach is based only on topological features of the network, not on the attributes of nodes, the applications of it are broader than the ones that are dataset-specific. Evolutionary algorithms integrating various niching techniques have been widely used to find multiple optima of an optimization problem. In recent years, an increasing amount of research has been focused on the design and application of speciation-based niching techniques. These techniques rely on speciation to partition a population into subpopulations (species) such that each occupies a different region of attraction (niche) on the fitness landscape. Existing speciation methods are either distance-based or topology-based. Topology-based methods are more flexible and have fewer assumptions than distance-based methods. However, existing topology-based methods all require sampling and evaluating new individuals in order to capture the landscape topography. This incurs additional fitness evaluations (FEs), which is a drawback, especially when the FE budget is limited. In this paper, a new topology-based speciation method named history-based topological speciation (HTS) is proposed. It relies exclusively on search history to capture the landscape topography and, therefore, does not require any additional FEs to be performed. To the best of our knowledge, HTS is the only parameter-free speciation method at the moment. Both theoretical and empirical analyses have been conducted. Theoretical analysis shows that HTS incurs acceptable computational overhead. In the experimental study, HTS outperformed existing topology-based methods on benchmark functions in up to 32-D space and with as many as 50 optima, and the time overhead was practically negligible if a single FE took seconds. Practical optimization problems frequently include uncertainty about the quality measure, for example due to noisy evaluations. Thus, they do not allow for a straightforward application of traditional optimization techniques. In these settings, randomized search heuristics such as evolutionary algorithms are a popular choice because they are often assumed to exhibit some kind of resistance to noise. Empirical evidence suggests that some algorithms, such as estimation of distribution algorithms (EDAs) are robust against a scaling of the noise intensity, even without resorting to explicit noise-handling techniques such as resampling. In this paper, we want to support such claims with mathematical rigor. We introduce the concept of graceful scaling in which the run time of an algorithm scales polynomially with noise intensity. We study a monotone fitness function over binary strings with additive noise taken from a Gaussian distribution. We show that myopic heuristics cannot efficiently optimize the function under arbitrarily intense noise without any explicit noise-handling. Furthermore, we prove that using a population does not help. Finally we show that a simple EDA called the Compact Genetic Algorithm can overcome the shortsightedness of mutation-only heuristics to scale gracefully with noise. We conjecture that recombinative genetic algorithms also have this property. This extended abstract summarizes our work "The Benefit of Recombination in Noisy Evolutionary Search," which appeared in Proceedings of International Symposium on Algorithms and Computation (ISAAC), 2015, pp. 140--150. This paper presents a novel evolutionary optimization strategy based on the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). This new approach is intended to reduce the number of generations required for convergence to the optimum. Reducing the number of generations, i.e., the time complexity of the algorithm, is important if a large population size is desired: (1) to reduce the effect of noise; (2) to improve global search properties; and (3) to implement the algorithm on (highly) parallel machines. Our method results in a highly parallel algorithm which scales favorably with large numbers of processors. This is accomplished by efficiently incorporating the available information from a large population, thus significantly reducing the number of generations needed to adapt the covariance matrix. The original version of the CMA-ES was designed to reliably adapt the covariance matrix in small populations but it cannot exploit large populations efficiently. Our modifications scale up the efficiency to population sizes of up to 10n, where n is the problem dimension. This method has been applied to a large number of test problems, demonstrating that in many cases the CMA-ES can be advanced from quadratic to linear time complexity.