<jupyter_start><jupyter_text>ARE YOU READY! 👏  I'm actually quite excited as this is a project as I have been wanting to do for a while...  Evolutionary Search over Deep Neural Network Architectures 🐣.  This is actually an old concept [1](https://ieeexplore.ieee.org/document/784219/) that has had a bit of a renaissance [2](https://arxiv.org/pdf/1703.00548.pdf) 📄.  Cool packages such as [DEVOL](https://github.com/joeddav/devol) implement a nice version of this for 2D Convoltuion, but I really wanted to do this myself, in order to implement this for other problems like Feed Forward Neural Networks or 1d Convolution 🤓.  [DEAP](http://deap.readthedocs.io/en/master/api/tools.html) is a popular python library for evolutionary computing and seems really versatile, so check it out.    Ok, time to import some libraries...*boring*📚.<jupyter_code>import random
import numpy as np

from deap import base, tools, creator

import tensorflow as tf
from keras.layers import Conv1D, MaxPooling1D
from keras.models import Model
from keras.layers import Input, Dense
from keras import regularizers

from sklearn.datasets import load_boston

import pandas as pd<jupyter_output>Using TensorFlow backend.<jupyter_text>We are going try the classic boston houing price dataset.  🍳<jupyter_code>data = load_boston()<jupyter_output><empty_output><jupyter_text>So the Genetic Algorithms involves four main steps...    __Initialize/Selection:__  Where we select new genes.  __Evalutation:__  Where we evaluate these genes.  __Cross-over:__  Where we 'breed' out individuals to exchange genes.  __Mutation:__  Where we introduce new genetic material.    Here, we let the number of neurons and the number of layers in our network be our genes and our loss function is the loss on our hold-out set.<jupyter_code>creator.create("FitnessMax", base.Fitness, weights=(-1.0,))
creator.create("Individual", list, fitness=creator.FitnessMax)<jupyter_output><empty_output><jupyter_text>As you can see here, regulatization is really important or the network can get too big, so I included regularization on each layer's regularization for backprop and then added a regularization term for Genetic Algorithm.<jupyter_code>def evalOneMax(individual, alpha=1/50):
    inputs = Input(shape=(data.data.shape[1],))
    
    dense1 = Dense(abs(individual[0]), 
                   kernel_regularizer=regularizers.l1_l2(0.1, 0.1), 
                   activation='relu')(inputs) if individual[0] != 0 else (inputs)
    
    dense2 = Dense(abs(individual[1]), 
                   kernel_regularizer=regularizers.l1_l2(0.1, 0.1), 
                   activation='relu')(inputs) if individual[1] != 0 else (dense1)
    
    dense3 = Dense(abs(individual[2]), 
                   kernel_regularizer=regularizers.l1_l2(0.1, 0.1), 
                   activation='relu')(inputs) if individual[2] != 0 else (dense2)
    
    output = Dense(1, kernel_regularizer=regularizers.l1_l2(0.1, 0.1), 
                   activation='softmax')(inputs)
    
    model = Model(inputs=inputs, outputs=output)
    
    model.compile('adam', loss=tf.nn.log_poisson_loss)
    model.fit(data.data, data.target,
                epochs=5,
                batch_size=20,
                shuffle=True,
                validation_split=False,
                verbose=0
           )
    return  (model.evaluate(data.data, data.target, batch_size=128, verbose=0).tolist() + alpha*sum(individual),)
toolbox = base.Toolbox()

N_ATTR = 3

# Attribute generator 
toolbox.register("attr_bool", random.randint,0,64)

# Structure initializers
toolbox.register("individual", tools.initRepeat, creator.Individual, toolbox.attr_bool, N_ATTR)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)

#Evolutionary Strategies
toolbox.register("evaluate", evalOneMax)
toolbox.register("mate", tools.cxTwoPoint)
toolbox.register("mutate", tools.mutUniformInt,low=0, up=64, indpb=0.3)
toolbox.register("select", tools.selTournament, tournsize=3)<jupyter_output><empty_output><jupyter_text>TIME TO EVOLVE!<jupyter_code>def main(N_POP=5, CXPB = 0.2, MUTPB = 0.5, GEN=10):
    """
    # N_POP is the population size
    # MUTPB is the probability for mutating an individual
    # CXPB  is the probability with which two individuals are crossed
    # GEN is the number of generations to be run by the algorithm
    """
    
    pop = toolbox.population(N_POP)
    
    # Evaluate the entire population
    fitnesses = list(map(toolbox.evaluate, pop))
    for ind, fit in zip(pop, fitnesses):
        ind.fitness.values = fit    
    
    # Extracting all the fitnesses of 
    fits = [ind.fitness.values[0] for ind in pop]

    # Variable keeping track of the number of generations
    g = 0
    
    # Begin the evolution
    while max(fits) < 5 and g < GEN:
        # A new generation
        g = g + 1
        print(f"-- Generation {g} --")
        # Select the next generation individuals
        offspring = toolbox.select(pop, len(pop))
        # Clone the selected individuals
        offspring = list(map(toolbox.clone, offspring))

        # Apply crossover and mutation on the offspring
        for child1, child2 in zip(offspring[::2], offspring[1::2]):
            if random.random() < CXPB:
                toolbox.mate(child1, child2)
                del child1.fitness.values
                del child2.fitness.values
        
        for mutant in offspring:
            if random.random() < MUTPB:
                toolbox.mutate(mutant)
              
        # Evaluate the individuals with an invalid fitness
        invalid_ind = [ind for ind in offspring if not ind.fitness.valid]
        fitnesses = map(toolbox.evaluate, invalid_ind)
        for ind, fit in zip(invalid_ind, fitnesses):
            ind.fitness.values = fit
              
        pop[:] = offspring
              
        # Gather all the fitnesses in one list and print the stats
        fits = [ind.fitness.values[0] for ind in pop]
        
        print(f'Offspring: {offspring}')
        print(f'Fitness Funcitons: {list(map(lambda x: round(x, 3),fits))}')
        
        length = len(pop)
        mean = sum(fits) / length
        sum2 = sum(x*x for x in fits)
        std = abs(sum2 / length - mean**2)**0.5
        
        print("  Min %s" % min(fits))
        print("  Max %s" % max(fits))
        print("  Avg %s" % mean)
        print("  Std %s" % std)
        
    return pop[fits.index(min(fits))]
pop = main(N_POP=10, GEN=10)<jupyter_output>WARNING: Logging before flag parsing goes to stderr.
W0807 11:53:07.238829 140093439174464 deprecation_wrapper.py:119] From /home/marcusskky/.conda/envs/Evolution/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

W0807 11:53:07.355266 140093439174464 deprecation_wrapper.py:119] From /home/marcusskky/.conda/envs/Evolution/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0807 11:53:07.376158 140093439174464 deprecation_wrapper.py:119] From /home/marcusskky/.conda/envs/Evolution/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W0807 11:53:07.491070 140093439174464 deprecation_wrapper.py:119] From /home/marcusskky/.conda/envs/Evolution/lib/python3.7/site-packages/keras/opt[...]<jupyter_text>This is awesome! Lets check it out.  What's really interesting is that this network has more neurons in the middle layer.<jupyter_code>print(pop)
loss = evalOneMax(pop, alpha=0)
print(loss)<jupyter_output>(-19.369102990674406,)
{'fraction_non_alphanumeric': 0.10177392205101644, 'fraction_numerical': 0.03729120807976175, 'mean_word_length': 3.253303964757709, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 22, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '197969', 'license': 'MIT', 'n_tokens_mistral': 2662, 'n_tokens_neox': 2405, 'n_words': 672, 'path': 'content/Evolving Neural Networks.ipynb', 'repo_name': 'marcusinthesky/DirtyData', 'size': '7723'}