Fitness functions based on test cases are very common in Genetic Programming (GP). This process can be assimilated to a learning task, with the inference of models from a limited number of samples. This paper is an investigation on two methods to improve generalization in GP-based learning: 1) the selection of the best-of-run individuals using a three data sets methodology, and 2) the application of parsimony pressure in order to reduce the complexity of the solutions. Results using GP in a binary classification setup show that while the accuracy on the test sets is preserved, with less variances compared to baseline results, the mean tree size obtained with the tested methods is significantly reduced.
Because of the bloat phenomenon, typical in GP, parsimony pressure has been more widely studied @cite_19 @cite_12 @cite_15 @cite_23 . In particular, several papers @cite_5 @cite_22 @cite_1 have produced interesting results around the idea of using a parsimony pressure to increase the generalization capability of GP-evolved solutions. However, a counter-argumentation is given in @cite_13 , where solutions biased toward low complexity have, in some circumstances, increased generalization error. This is in accordance with the argumentation given in @cite_17 , which states that less complex solutions are not always more robust.
@cite_22 @cite_1 @cite_19 @cite_23 @cite_5 @cite_15 @cite_13 @cite_12 @cite_17
2803852436 1982359839 2763281404 2962919088
For theoretical analyses there are two specifics distinguishing GP from many other areas of evolutionary computation. First, the variable size representations, in particular yielding a possible bloat (i.e. the growth of individuals with redundant parts). Second, the role and realization of crossover, which is particularly central in GP due to the tree-based representation. Whereas some theoretical work on GP has studied the effects of bloat, crossover had a surprisingly little share in this work. We propose and analyze a discontinuous Galerkin approximation for the Stokes problem. The finite element triangulation employed is not required to be conforming and we use discontinuous pressures and velocities. No additional unknown fields need to be introduced, but only suitable bilinear forms defined on the interfaces between the elements, involving the jumps of the velocity and the average of the pressure. We consider hp approximations using ℚk′–ℚk velocity-pressure pairs with k′ = k + 2, k + 1, k. Our methods show better stability properties than the corresponding conforming ones. We prove that our first two choices of velocity spaces ensure uniform divergence stability with respect to the mesh size h. Numerical results show that they are uniformly stable with respect to the local polynomial degree k, a property that has no analog in the conforming case. An explicit bound in k which is not sharp is also proven. Numerical results show that if equal order approximation is chosen for the velocity and pressure, no spurious pressure modes are present but the method is not uniformly stable either with respect to h or k. We derive a priori error estimates generalizing the abstract theory of mixed methods. Optimal error estimates in h are proven. As for discontinuous Galerkin methods for scalar diffusive problems, half of the power of k is lost for p and hp pproximations independently of the divergence stability. When solving consensus optimization problems over a graph, there is often an explicit characterization of the convergence rate of Gradient Descent (GD) using the spectrum of the graph Laplacian. The same type of problems under the Alternating Direction Method of Multipliers (ADMM) are, however, poorly understood. For instance, simple but important non-strongly-convex consensus problems have not yet being analyzed, especially concerning the dependency of the convergence rate on the graph topology. Recently, for a non-strongly-convex consensus problem, a connection between distributed ADMM and lifted Markov chains was proposed, followed by a conjecture that ADMM is faster than GD by a square root factor in its convergence time, in close analogy to the mixing speedup achieved by lifting several Markov chains. Nevertheless, a proof of such a claim is is still lacking. Here we provide a full characterization of the convergence of distributed over-relaxed ADMM for the same type of consensus problem in terms of the topology of the underlying graph. Our results provide explicit formulas for optimal parameter selection in terms of the second largest eigenvalue of the transition matrix of the graph's random walk. Another consequence of our results is a proof of the aforementioned conjecture, which interestingly, we show it is valid for any graph, even the ones whose random walks cannot be accelerated via Markov chain lifting. We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.