Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> In real-world problems related to finance, business, and management, mathematicians and economists frequently encounter optimization problems. In this classic book, George Dantzig looks at a wealth of examples and develops linear programming methods for their solutions. He begins by introducing the basic theory of linear inequalities and describes the powerful simplex method used to solve them. Treatments of the price concept, the transportation problem, and matrix methods are also given, and key mathematical concepts such as the properties of convex sets and linear vector spaces are covered."The author of this book was the main force in establishing a new mathematical discipline, and he has contributed to its further development at every stage and from every angle. This volume ... is a treasure trove for those who work in this field--teachers, students, and users alike. Its encyclopaedic coverage, due in part to collaboration with other experts, makes it an absolute must."--S. Vajda, Zentralblatt fYr Mathematik und ihre Grenzgebiete <s> BIB001 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> There is a deep and useful connection between statistical mechanics (the behavior of systems with many degrees of freedom in thermal equilibrium at a finite temperature) and multivariate or combinatorial optimization (finding the minimum of a given function depending on many parameters). A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems. This connection to statistical mechanics exposes new information and provides an unfamiliar perspective on traditional optimization problems and methods. <s> BIB002 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> A concept for the optimization of nonlinear functions using particle swarm methodology is introduced. The evolution of several paradigms is outlined, and an implementation of one of the paradigms is discussed. Benchmark testing of the paradigm is described, and applications, including nonlinear function optimization and neural network training, are proposed. The relationships between particle swarm optimization and both artificial life and genetic algorithms are described. <s> BIB003 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> This paper discusses the trade-off between accuracy, reliability and computing time in global optimization. Particular compromises provided by traditional methods (Quasi-Newton and Nelder-Mead's simplex methods) and genetic algorithms are addressed and illustrated by a particular application in the field of nonlinear system identification. Subsequently, new hybrid methods are designed, combining principles from genetic algorithms and "hill-climbing" methods in order to find a better compromise to the trade-off. Inspired by biology and especially by the manner in which living beings adapt themselves to their environment, these hybrid methods involve two interwoven levels of optimization, namely evolution (genetic algorithms) and individual learning (Quasi-Newton), which cooperate in a global process of optimization. One of these hybrid methods appears to join the group of state-of-the-art global optimization methods: it combines the reliability properties of the genetic algorithms with the accuracy of Quasi-Newton method, while requiring a computation time only slightly higher than the latter. <s> BIB004 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> A new heuristic approach for minimizing possibly nonlinear and non-differentiable continuous space functions is presented. By means of an extensive testbed it is demonstrated that the new method converges faster and with more certainty than many other acclaimed global optimization methods. The new method requires few control variables, is robust, easy to use, and lends itself very well to parallel computation. <s> BIB005 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> We propose a hybrid algorithm for finding a set of nondominated solutions of a multi objective optimization problem. In the proposed algorithm, a local search procedure is applied to each solution (i.e., each individual) generated by genetic operations. Our algorithm uses a weighted sum of multiple objectives as a fitness function. The fitness function is utilized when a pair of parent solutions are selected for generating a new solution by crossover and mutation operations. A local search procedure is applied to the new solution to maximize its fitness value. One characteristic feature of our algorithm is to randomly specify weight values whenever a pair of parent solutions are selected. That is, each selection (i.e., the selection of two parent solutions) is performed by a different weight vector. Another characteristic feature of our algorithm is not to examine all neighborhood solutions of a current solution in the local search procedure. Only a small number of neighborhood solutions are examined to prevent the local search procedure from spending almost all available computation time in our algorithm. High performance of our algorithm is demonstrated by applying it to multi objective flowshop scheduling problems. <s> BIB006 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Evolutionary programming (EP) has been applied with success to many numerical and combinatorial optimization problems in recent years. EP has rather slow convergence rates, however, on some function optimization problems. In the paper, a "fast EP" (FEP) is proposed which uses a Cauchy instead of Gaussian mutation as the primary search operator. The relationship between FEP and classical EP (CEP) is similar to that between fast simulated annealing and the classical version. Both analytical and empirical studies have been carried out to evaluate the performance of FEP and CEP for different function optimization problems. The paper shows that FEP is very good at search in a large neighborhood while CEP is better at search in a small local neighborhood. For a suite of 23 benchmark problems, FEP performs much better than CEP for multimodal functions with many local minima while being comparable to CEP in performance for unimodal and multimodal functions with only a few local minima. The paper also shows the relationship between the search step size and the probability of finding a global optimum and thus explains why FEP performs better than CEP on some functions but not on others. In addition, the importance of the neighborhood size and its relationship to the probability of finding a near-optimum is investigated. Based on these analyses, an improved FEP (IFEP) is proposed and tested empirically. This technique mixes different search operators (mutations). The experimental results show that IFEP performs better than or as well as the better of FEP and CEP for most benchmark problems tested. <s> BIB007 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Hybrid metaheuristics have received considerable interest these recent years in the field of combinatorial optimization. A wide variety of hybrid approaches have been proposed in the literature. In this paper, a taxonomy of hybrid metaheuristics is presented in an attempt to provide a common terminology and classification mechanisms. The taxonomy, while presented in terms of metaheuristics, is also applicable to most types of heuristics and exact optimization algorithms. ::: ::: As an illustration of the usefulness of the taxonomy an annoted bibliography is given which classifies a large number of hybrid approaches according to the taxonomy. <s> BIB008 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> List of Figures. List of Tables. Preface. Contributing Authors. Series Foreword. Part I: Foundations. 1. An Introduction to Evolutionary Algorithms J.A. Lozano. 2. An Introduction to Probabilistic Graphical Models P. Larranaga. 3. A Review on Estimation of Distribution Algorithms P. Larranaga. 4. Benefits of Data Clustering in Multimodal Function Optimization via EDAs J.M. Pena, et al. 5. Parallel Estimation of Distribution Algorithms J.A. Lozano, et al. 6. Mathematical Modeling of Discrete Estimation of Distribution Algorithms C. Gonzalez, et al. Part II: Optimization. 7. An Empiricial Comparison of Discrete Estimation of Distribution Algorithms R. Blanco., J.A. Lozano. 8. Results in Function Optimization with EDAs in Continuous Domain E. Bengoetxea, et al. 9. Solving the 0-1 Knapsack Problem with EDAs R. Sagarna, P. Larranaga. 10. Solving the Traveling Salesman Problem with EDAs V. Robles, et al. 11. EDAs Applied to the Job Shop Scheduling Problem J.A. Lozano, A. Mendiburu. 12. Solving Graph Matching with EDAs Using a Permutation-Based Representation E. Bengoetxea, et al. Part III: Machine Learning. 13. Feature Subset Selection by Estimation of Distribution Algorithms I. Inza, et al. 14. Feature Weighting for Nearest Neighbor by EDAs I. Inza, et al. 15. Rule Induction by Estimation of Distribution Algorithms B. Sierra, et al. 16. Partial Abductive Inference in Bayesian Networks: An Empirical Comparison Between GAs and EDAs L.M. de Campos, et al.17. Comparing K-Means, GAs and EDAs in Partitional Clustering J. Roure, et al. 18. Adjusting Weights in Artificial Neural Networks using Evolutionary Algorithms C. Cotta, et al. Index. <s> BIB009 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Abstract Simulated annealing is a naturally serial algorithm, but its behavior can be controlled by the cooling schedule. Genetic algorithm exhibits implicit parallelism and can retain useful redundant information about what is learned from previous searches by its representation in individuals in the population, but GA may lose solutions and substructures due to the disruptive effects of genetic operators and is not easy to regulate GA's convergence. By reasonably combining these two global probabilistic search algorithms, we develop a general, parallel and easily implemented hybrid optimization framework, and apply it to job-shop scheduling problems. Based on effective encoding scheme and some specific optimization operators, some benchmark job-shop scheduling problems are well solved by the hybrid optimization strategy, and the results are competitive with the best literature results. Besides the effectiveness and robustness of the hybrid strategy, the combination of different search mechanisms and structures can relax the parameter-dependence of GA and SA. Scope and purpose Job-shop scheduling problem (JSP) is one of the most well-known machine scheduling problems and one of the strongly NP-hard combinatorial optimization problems. Developing effective search methods is always an important and valuable work. The scope and purpose of this paper is to present a parallel and easily implemented hybrid optimization framework, which reasonably combines genetic algorithm with simulated annealing. Based on effective encoding scheme and some specific optimization operators, the job-shop scheduling problems are well solved by the hybrid optimization strategy. <s> BIB010 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> The field of metaheuristics for the application to combinatorial optimization problems is a rapidly growing field of research. This is due to the importance of combinatorial optimization problems for the scientific as well as the industrial world. We give a survey of the nowadays most important metaheuristics from a conceptual point of view. We outline the different components and concepts that are used in the different metaheuristics in order to analyze their similarities and differences. Two very important concepts in metaheuristics are intensification and diversification. These are the two forces that largely determine the behavior of a metaheuristic. They are in some way contrary but also complementary to each other. We introduce a framework, that we call the I&D frame, in order to put different intensification and diversification components into relation with each other. Outlining the advantages and disadvantages of different metaheuristic approaches we conclude by pointing out the importance of hybridization of metaheuristics as well as the integration of metaheuristics and other methods for optimization. <s> BIB011 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Manifold possibilities of hybridizing individual metaheuristics with each other and/or with algorithms from other fields exist. A large number of publications documents the benefits and great success of such hybrids. This article overviews several popular hybridization approaches and classifies them based on various characteristics. In particular with respect to low-level hybrids of different metaheuristics, a unified view based on a common pool template is described. It helps in making similarities and different key components of existing metaheuristics explicit. We then consider these key components as a toolbox for building new, effective hybrid metaheuristics. This approach of thinking seems to be superior to sticking too strongly to the philosophies and historical backgrounds behind the different metaheuristic paradigms. Finally, particularly promising possibilities of combining metaheuristics with constraint programming and integer programming techniques are highlighted. <s> BIB012 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> A hybrid particle swarm with differential evolution operator, termed DEPSO, which provide the bell-shaped mutations with consensus on the population diversity along with the evolution, while keeping the self-organized particle swarm dynamics, is proposed. Then it is applied to a set of benchmark functions, and the experimental results illustrate its efficiency. <s> BIB013 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> An evolutionary recurrent network which automates the design of recurrent neural/fuzzy networks using a new evolutionary learning algorithm is proposed in this paper. This new evolutionary learning algorithm is based on a hybrid of genetic algorithm (GA) and particle swarm optimization (PSO), and is thus called HGAPSO. In HGAPSO, individuals in a new generation are created, not only by crossover and mutation operation as in GA, but also by PSO. The concept of elite strategy is adopted in HGAPSO, where the upper-half of the best-performing individuals in a population are regarded as elites. However, instead of being reproduced directly to the next generation, these elites are first enhanced. The group constituted by the elites is regarded as a swarm, and each elite corresponds to a particle within it. In this regard, the elites are enhanced by PSO, an operation which mimics the maturing phenomenon in nature. These enhanced elites constitute half of the population in the new generation, whereas the other half is generated by performing crossover and mutation operation on these enhanced elites. HGAPSO is applied to recurrent neural/fuzzy network design as follows. For recurrent neural network, a fully connected recurrent neural network is designed and applied to a temporal sequence production problem. For recurrent fuzzy network design, a Takagi-Sugeno-Kang-type recurrent fuzzy network is designed and applied to dynamic plant control. The performance of HGAPSO is compared to both GA and PSO in these recurrent networks design problems, demonstrating its superiority. <s> BIB014 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Over the last decade, memetic algorithms (MAs) have relied on the use of a variety of different methods as the local improvement procedure. Some recent studies on the choice of local search method employed have shown that this choice significantly affects the efficiency of problem searches. Given the restricted theoretical knowledge available in this area and the limited progress made on mitigating the effects of incorrect local search method choice, we present strategies for MA control that decide, at runtime, which local method is chosen to locally improve the next chromosome. The use of multiple local methods during a MA search in the spirit of Lamarckian learning is here termed Meta-Lamarckian learning. Two adaptive strategies for Meta-Lamarckian learning are proposed in this paper. Experimental studies with Meta-Lamarckian learning strategies on continuous parametric benchmark problems are also presented. Further, the best strategy proposed is applied to a real-world aerodynamic wing design problem and encouraging results are obtained. It is shown that the proposed approaches aid designers working on complex engineering problems by reducing the probability of employing inappropriate local search methods in a MA, while at the same time, yielding robust and improved design search performance. <s> BIB015 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> The combination of evolutionary algorithms with local search was named "memetic algorithms" (MAs) (Moscato, 1989). These methods are inspired by models of natural systems that combine the evolutionary adaptation of a population with individual learning within the lifetimes of its members. Additionally, MAs are inspired by Richard Dawkin's concept of a meme, which represents a unit of cultural evolution that can exhibit local refinement (Dawkins, 1976). In the case of MA's, "memes" refer to the strategies (e.g., local refinement, perturbation, or constructive methods, etc.) that are employed to improve individuals. In this paper, we review some works on the application of MAs to well-known combinatorial optimization problems, and place them in a framework defined by a general syntactic model. This model provides us with a classification scheme based on a computable index D, which facilitates algorithmic comparisons and suggests areas for future research. Also, by having an abstract model for this class of metaheuristics, it is possible to explore their design space and better understand their behavior from a theoretical standpoint. We illustrate the theoretical and practical relevance of this model and taxonomy for MAs in the context of a discussion of important design issues that must be addressed to produce effective and efficient MAs. <s> BIB016 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Abstract As a novel optimization technique, chaos has gained much attention and some applications during the past decade. For a given energy or cost function, by following chaotic ergodic orbits, a chaotic dynamic system may eventually reach the global optimum or its good approximation with high probability. To enhance the performance of particle swarm optimization (PSO), which is an evolutionary computation technique through individual improvement plus population cooperation and competition, hybrid particle swarm optimization algorithm is proposed by incorporating chaos. Firstly, adaptive inertia weight factor (AIWF) is introduced in PSO to efficiently balance the exploration and exploitation abilities. Secondly, PSO with AIWF and chaos are hybridized to form a chaotic PSO (CPSO), which reasonably combines the population-based evolutionary searching ability of PSO and chaotic searching behavior. Simulation results and comparisons with the standard PSO and several meta-heuristics show that the CPSO can effectively enhance the searching efficiency and greatly improve the searching quality. <s> BIB017 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Differential evolution (DE) was very successful in solving the global continuous optimization problem. It mainly uses the distance and direction information from the current population to guide its further search. Estimation of distribution algorithm (EDA) samples new solutions from a probability model which characterizes the distribution of promising solutions. This paper proposes a combination of DE and EDA (DE/EDA) for the global continuous optimization problem. DE/EDA combines global information extracted by EDA with differential information obtained by DE to create promising solutions. DE/EDA has been compared with the best version of the DE algorithm and an EDA on several commonly utilized test problems. Experimental results demonstrate that DE/EDA outperforms the DE algorithm and the EDA. The effect of the parameters of DE/EDA to its performance is investigated experimentally. <s> BIB018 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> The ten papers in this special section are devoted to memetic algorithms. The papers are loosely grouped into two categories: memetic algorithm methodologies and domain-specific memetic algorithms. Briefly summarizes the articles included in this section. <s> BIB019 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Optimization problems are of great importance in many fields. They can be tackled, for example, by approximate algorithms such as metaheuristics. Examples of metaheuristics are simulated annealing, tabu search, evolutionary computation, iterated local search, variable neighborhood search, and ant colony optimization. In recent years it has become evident that a skilled combination of a metaheuristic with other optimization techniques, a so called hybrid metaheuristic, can provide a more efficient behavior and a higher flexibility. This is because hybrid metaheuristics combine their advantages with the complementary strengths of, for example, more classical optimization techniques such as branch and bound or dynamic programming. The authors involved in this book are among the top researchers in their domain. The book is intended both to provide an overview of hybrid metaheuristics to novices of the field, and to provide researchers from the field with a collection of some of the most interesting recent developments. <s> BIB020 </s> Hybridizing Differential Evolution and Particle Swarm Optimization to Design Powerful Optimizers: A Review and Taxonomy <s>  <s> Swarm intelligence is a relatively new approach to problem solving that takes inspiration from the social behaviors of insects and of other animals. In particular, ants have inspired a number of methods and techniques among which the most studied and the most successful is the general purpose optimization technique known as ant colony optimization. Ant colony optimization (ACO) takes inspiration from the foraging behavior of some ant species. These ants deposit pheromone on the ground in order to mark some favorable path that should be followed by other members of the colony. Ant colony optimization exploits a similar mechanism for solving optimization problems. From the early nineties, when the first ant colony optimization algorithm was proposed, ACO attracted the attention of increasing numbers of researchers and many successful applications are now available. Moreover, a substantial corpus of theoretical results is becoming available that provides useful guidelines to researchers and practitioners in further applications of ACO. The goal of this article is to introduce ant colony optimization and to survey its most notable applications <s> BIB021
cost. Besides, the practical problems are vastly varied so that it is hard to find a universal optimizer without any changes to best solve all problems. The diversity of problems contributes to the diversity of optimizers to a large extent. Therefore, it is not unusual to see that numerous optimizers have emerged since Dantzig proposed the simplex method for the linear programming problem in 1947 as an inception of optimization research BIB001 . Now, increasingly, problems turn out to be nonlinear, nonconvex, multimodal, discontinuous, and even dynamic. For these complicated problems, stochastic optimizers become favored as they have no dependence on the differentiability, continuity, and convexity of objective functions. Besides, many stochastic optimizers have good performance in global optimization. In the family of stochastic optimizers, well-known typical optimizers include the genetic algorithm (GA)  , evolutionary strategy (ES)  (CMA-ES  in particular), evolutionary programming (EP) BIB007 , simulated annealing (SA) BIB002 , ant-colony optimization (ACO) BIB021 , particle swarm optimization (PSO) BIB003 , differential evolution (DE) BIB005 , estimation of distribution algorithm (EDA) BIB009 , and so on. All of them have many variants, which have excellent performance. These variants are based on various improvement strategies. Hybridization is one of the most efficient strategies to improve the performance of many optimizers BIB020    BIB008 . In genetics, hybridization is the process to combine different varieties or species of organisms to create a hybrid (biology). In evolutionary algorithms (EAs), hybridization refers to merging two or more optimization techniques into a single algorithm. In the research field of combinatorial optimization, hybrid optimizers are also termed as hybrid metaheuristics BIB008  BIB011 BIB012 . In the past decade, hybrid optimizers have attracted persistent attention from scholars that are interested in design of optimizers and their applications BIB020    BIB008  BIB011 BIB012 . As Raidl claimed in his unified view of hybrid metaheuristics BIB012 , it seems that choosing an adequate hybrid approach is determinant to achieve top performance in solving most difficult problems. Like the crossover operator in GAs  , hybridization contributes to population diversity but the population here means the family of optimizers. There are many successful examples of hybridization in the evolution process of this family (e.g., BIB016 BIB014 BIB004 BIB006 BIB017 BIB015 BIB010  BIB018 BIB013 ). A common template for hybridization is provided by memetic algorithms (MAs), which combine the respective advantages of global search and local search (LS) BIB016 . Due to excellent performance, MAs have been favored by many scholars in their research on different optimization problems BIB019 . However, MAs only represent a special class in the family of hybrid optimizers. There are manifold possibilities of hybridizing different optimizers, which follow diverse philosophies and paradigms. This review paper aims to establish a comprehensive taxonomy of hybridization strategies by the review of existing hybrid optimizers that are based on two powerful POs-DE and PSO. To build a unified nomenclature, we term any optimizer that is based on hybridization between DE and PSO as DEPSO. To date, it is impossible to list all kinds of hybrids within a paper of tens of pages, since there are too many hybrids that exist in the literature (about 1450 000 related works according to the Google search result with the index terms "hybrid algorithm" and "optimization"). One reason we choose DEPSO as the representative for hybrid optimizers to build the taxonomy is that both PSO and DE are POs that involve masses of hybridization factors. Besides, a large number of DEPSO variants have been proposed in the literature, laying a solid foundation for the establishment of a comprehensive taxonomy. Moreover, as pointed out later, the proposed taxonomy that is based on DEPSO can be easily applied to other hybrid optimizers. In order to provide a comprehensive reference to design powerful hybrid optimizers, we pool diverse ideas of scholars who have contributed to DEPSO's development. The contribution of this paper is that it provides readers not only a review on hybridization of DE and PSO, but more importantly a broader sight and a tool on how to differentiate and design different hybrid optimizers by a comprehensive systematic taxonomy of hybridization strategies. This paper is structured as follows. In Section II, a brief introduction on PSO and DE is presented. In Section III, previous works on DEPSO are reviewed. In Section IV, a new taxonomy of hybridization strategies is proposed in an attempt to provide a common terminology and classification mechanisms. Most existing DEPSOs to date and other typical hybrid optimizers are categorized according to the taxonomy. In Section V, we present a discussion about some common issues of hybrid optimizers (not restricted to DEPSO) and indicate a number of topics that are worthy of future research. In Section VI, we conclude the paper.