Recently, there emerged revived interests of designing automatic programs (e.g., using genetic evolutionary algorithms) to optimize the structure of Convolutional Neural Networks (CNNs) for a specific task. The challenge in designing such programs lies in how to balance between large search space of the network structures and high computational costs. Existing works either impose strong restrictions on the search space or use enormous computing resources. In this paper, we study how to design a genetic programming approach for optimizing the structure of a CNN for a given task under limited computational resources yet without imposing strong restrictions on the search space. To reduce the computational costs, we propose two general strategies that are observed to be helpful: (i) aggressively selecting strongest individuals for survival and reproduction, and killing weaker individuals at a very early age; (ii) increasing mutation frequency to encourage diversity and faster evolution. The combined strategy with additional optimization techniques allows us to explore a large search space but with affordable computational costs. Our results on standard benchmark datasets (MNIST, SVHN, CIFAR-10, CIFAR-100) are competitive to similar approaches with significantly reduced computational costs.
The focus of @cite_4 is to scale up neuro-evolution to take advantage of the tremendous computational resources Google LLC has. A total of 250 GPUs are used in their experiments. They used a graph to encode an individual, and defined seven mutations to change the structure of networks they also used several other mutations that do not affect the structure of the network. , and used a standard binary tournament selection method @cite_15 for selecting individuals for reproduction.
@cite_15 @cite_4
2552732996 2038794597 2884711234 2004564253
Nowadays, in the world of limited attention, the techniques that maximize the spread of social influence are more than welcomed. Companies try to maximize their profits on sales by providing customers with free samples believing in the power of word-of-mouth marketing, governments and non-governmental organizations often want to introduce positive changes in the society by appropriately selecting individuals or election candidates want to spend least budget yet still win the election. In this work we propose the use of evolutionary algorithm as a mean for selecting seeds in social networks. By framing the problem as genetic algorithm challenge we show that it is possible to outperform well-known greedy algorithm in the problem of influence maximization for the linear threshold model in both: quality (up to 16 better) and efficiency (up to 35 times faster). We implemented these two algorithms by using GPGPU approach showing that also the evolutionary algorithm can benefit from GPU acceleration making it efficient and scaling better than the greedy algorithm. As the experiments conducted by using three real world datasets reveal, the evolutionary approach proposed in this paper outperforms the greedy algorithm in terms of the outcome and it also scales much better than the greedy algorithm when the network size is increasing. The only drawback in the GPGPU approach so far is the maximum size of the network that can be processed - it is limited by the memory of the GPU card. We believe that by showing the superiority of the evolutionary approach over the greedy algorithm, we will motivate the scientific community to look for an idea to overcome this limitation of the GPU approach - we also suggest one of the possible paths to explore. Since the proposed approach is based only on topological features of the network, not on the attributes of nodes, the applications of it are broader than the ones that are dataset-specific. The idea of using evolutionary computation to train artificial neural networks, or neuroevolution (NE), for reinforcement learning (RL) tasks has now been around for over 20 years. However, as RL tasks become more challenging, the networks required become larger, as do their genomes. But, scaling NE to large nets (i.e. tens of thousands of weights) is infeasible using direct encodings that map genes one-to-one to network components. In this paper, we scale-up our compressed network encoding where network weight matrices are represented indirectly as a set of Fourier-type coefficients, to tasks that require very-large networks due to the high-dimensionality of their input space. The approach is demonstrated successfully on two reinforcement learning tasks in which the control networks receive visual input: (1) a vision-based version of the octopus control task requiring networks with over 3 thousand weights, and (2) a version of the TORCS driving game where networks with over 1 million weights are evolved to drive a car around a track using video images from the driver's perspective. Synchronized stochastic gradient descent (SGD) optimizers with data parallelism are widely used in training large-scale deep neural networks. Although using larger mini-batch sizes can improve the system scalability by reducing the communication-to-computation ratio, it may hurt the generalization ability of the models. To this end, we build a highly scalable deep learning training system for dense GPU clusters with three main contributions: (1) We propose a mixed-precision training method that significantly improves the training throughput of a single GPU without losing accuracy. (2) We propose an optimization approach for extremely large mini-batch size (up to 64k) that can train CNN models on the ImageNet dataset without losing accuracy. (3) We propose highly optimized all-reduce algorithms that achieve up to 3x and 11x speedup on AlexNet and ResNet-50 respectively than NCCL-based training on a cluster with 1024 Tesla P40 GPUs. On training ResNet-50 with 90 epochs, the state-of-the-art GPU-based system with 1024 Tesla P100 GPUs spent 15 minutes and achieved 74.9 top-1 test accuracy, and another KNL-based system with 2048 Intel KNLs spent 20 minutes and achieved 75.4 accuracy. Our training system can achieve 75.8 top-1 test accuracy in only 6.6 minutes using 2048 Tesla P40 GPUs. When training AlexNet with 95 epochs, our system can achieve 58.7 top-1 test accuracy within 4 minutes, which also outperforms all other existing systems. Abstract We previously introduced a voxel-based, multi-modal application of the partial least square algorithm (MMPLS) to characterize the linkage between patterns in a person's complementary complex datasets without the need to correct for multiple regional comparisons. Here we used it to demonstrate a strong correlation between MMPLS scores to characterize the linkage between the covarying patterns of fluorodeoxyglucose positron emission tomography (FDG PET) measurements of regional glucose metabolism and magnetic resonance imaging (MRI) measurements of regional gray matter associated with apolipoprotein E (APOE) e4 gene dose (i.e., three levels of genetic risk for late-onset Alzheimer's disease (AD)) in cognitively normal, late-middle-aged persons. Coregistered and spatially normalized FDG PET and MRI images from 70 of the subjects (27 e4 homozygotes, 36 e4 heterozygotes and 67 e4 non-carriers) were used in a hypothesis-generating MMPLS analysis to characterize the covarying pattern of regional gray matter volume and cerebral glucose metabolism most strongly correlated with APOE-e4 gene dose. Coregistered and spatially normalized FDG PET and MRI images from the remaining 30 of the subjects were used in a hypothesis-testing MMPLS analysis to generate FDG PET-MRI gray matter MMPLS scores blind to their APOE genotype and characterize their relationship to APOE-e4 gene dose. The hypothesis-generating analysis revealed covarying regional gray matter volume and cerebral glucose metabolism patterns that resembled those in traditional univariate analyses of AD and APOE-e4 gene dose and PET-MRI scores that were strongly correlated with APOE-e4 gene dose (p