<jupyter_start><jupyter_text>Finding best ANN Topology using pygeneticIn this example we are going to walk through the usage of GAEngine to find best ANN topology to solve a given classification problem.  1. Chromosome Representation A chromosome to represent different network topology parameters such as number of neurons in each layer, activation functions used and optimization algorithm used to train the neural network is used  For example,For a 2 hidden layer neural network, a chromosome can be represented as [2,”sigmoid”,3,”sigmoid”, “relu”, “adam” ] which means first hidden layer has two neurons and sigmoid activation function, second layer has three neurons and sigmoid activation function while the output layer has “relu” activation. The neural network is trained using Adam optimization. This can be easily done by defining a new ChromosomeFactory which takes the following as input* `hiddenLayers` = the number of hidden layers in the ANN* `neuronsPerLayer` = a list of all possible neurons per layer values the GA should include in it's search* `activations` = a list of all possible activation functions the GA should include in it's search* `optimizers` = a list of all possible ANN optimizer values the GA should include in it's searchIt should output `(noOfNeurons,activation_function`) pairs for each hidden layer followed by `output_activation` and `optimizer`<jupyter_code>from pygenetic.ChromosomeFactory import ChromosomeFactory
import random
class ANNTopologyChromosomeFactory(ChromosomeFactory):
    def __init__(self,hiddenLayers,neuronsPerLayer,activations,optimizers):
        self.neuronsPerLayer = neuronsPerLayer
        self.activations = activations
        self.optimizers = optimizers
        self.hiddenLayers = hiddenLayers

    def createChromosome(self):
        chromosome = []
        for i in range(self.hiddenLayers):
            chromosome.extend([random.choice(self.neuronsPerLayer), random.choice(self.activations)])
        chromosome.append(random.choice(self.activations))
        chromosome.append(random.choice(self.optimizers))
        return chromosome<jupyter_output><empty_output><jupyter_text>Lets solve the ANN topology problem for * `hiddenLayers` = 2* `neuronsPerLayer` = [2,5,10,12]* `activations` = ['relu','sigmoid']* `optimizers` = ['adam','sgd']Let's create this factory<jupyter_code>hidden_layers=2
neuronsPerLayer=[2,5,10,12]
activations=['relu','sigmoid']
optimizers=['adam','sgd']

factory = ANNTopologyChromosomeFactory(hiddenLayers=hidden_layers,neuronsPerLayer=neuronsPerLayer,activations=activations,
                                       optimizers=optimizers)<jupyter_output><empty_output><jupyter_text>You can also check if the factory works as expected by calling createChromosome function and observing the chromosome produced by the factory<jupyter_code># Code to test if factory works as expected
for i in range(5):
    print('Chromosome created: ', factory.createChromosome())<jupyter_output>Chromosome created:  [5, 'sigmoid', 2, 'sigmoid', 'relu', 'sgd']
Chromosome created:  [5, 'relu', 5, 'sigmoid', 'sigmoid', 'sgd']
Chromosome created:  [10, 'relu', 2, 'sigmoid', 'relu', 'sgd']
Chromosome created:  [5, 'relu', 5, 'sigmoid', 'relu', 'adam']
Chromosome created:  [10, 'relu', 10, 'sigmoid', 'sigmoid', 'sgd']<jupyter_text>2. Fitness function Fitness for a given chromosome is the loss of the neural network of topology corresponding to that chromosome when trained for a given amount of time. So lesser the fitness, the better the chromosome. Hence our GA is a minimization problem.<jupyter_code>from keras.models import Sequential
from keras.layers import Dense
def fitness(chromosome,X,Y,hidden_layers,input_dim,loss,metrics,epochs,batch_size):
    # create model
    model = Sequential()
    model.add(Dense(chromosome[0], input_dim= input_dim, activation=chromosome[1]))
    for i in range(hidden_layers-1):
        model.add(Dense(chromosome[i+2], activation=chromosome[i+3]))
    model.add(Dense(1, activation=chromosome[len(chromosome)-2]))
    # Compile model
    model.compile(loss=loss, optimizer=chromosome[len(chromosome)-1], metrics=[metrics])
    # Fit the model
    model.fit(X, Y, epochs=epochs, batch_size=batch_size,verbose=1)
    # evaluate the model
    scores = model.evaluate(X, Y)
    return scores[0]<jupyter_output>/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.<jupyter_text>We need then create a `GAEngine` instance from the `pygenetic` package and set the following* `factory` = the ANNTopologyChromosomeFactory factory instance we had intially created* `population_size = 10` would be a good number for this problem ( for demonstration purposes)* `cross_prob = 0.7`* `mut_prob = 0.2`* `fitness_type = 'min'` since this GA is a minimization problem<jupyter_code>from pygenetic import GAEngine
ga = GAEngine.GAEngine(factory,population_size=20,fitness_type='min',cross_prob = 0.7,mut_prob = 0.2)<jupyter_output><empty_output><jupyter_text>Lets now import our input data for the ANN and add the fitness function we had defined to this `GAEngine` instance<jupyter_code>import numpy
# load pima indians dataset
dataset = numpy.loadtxt("input.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]<jupyter_output><empty_output><jupyter_text>Lets use `binary_crossentropy` and `accuracy` for loss and metrics respectively<jupyter_code>ga.setFitnessHandler(fitness,X,Y,hidden_layers,len(X[0]),'binary_crossentropy','accuracy',30,10)<jupyter_output><empty_output><jupyter_text>3. Determing other attributes of the GA Many Standard Crossover, Mutation, Selection and Fitness functions are present in the `Utils` module of the `pygenetic` package.<jupyter_code>from pygenetic import Utils<jupyter_output><empty_output><jupyter_text>The use of popular crossover methods such as 1-point, 2-point crossover, etc can be used. Every child produced by such a crossover represents a new neural network topology which has the potential to possibly be better than others observed so far.  This can be done using the addCrossoverHandler of the pygenetic module which takes as parameters* crossover_function = the crossover function to be used* weight = the weightage the crossover function needs to be given (mainly used when multiple crossovers are added)<jupyter_code>ga.addCrossoverHandler(Utils.CrossoverHandlers.onePoint,1)<jupyter_output><empty_output><jupyter_text>Mutation The use of mutations such as flipping an attribute to another of its possible values, swapping values between two same attributes of the chromosome. Each such child would represent a small change to a previous neural network which could potentially better than others observed so far.We can define this function easily<jupyter_code>import copy
def mutation(chromosome,neuronsPerLayer,activations,optimizers):
    r = random.randint(0,len(chromosome)-1)
    newchrom = copy.copy(chromosome)
    if r == len(chromosome) - 1:
        newchrom[r] = random.choice(optimizers)
    elif r%2 == 1 or r == len(chromosome)-2:
        newchrom[r] = random.choice(activations)
    elif r%2 == 0:
        newchrom[r] = random.choice(neuronsPerLayer)
    return newchrom<jupyter_output><empty_output><jupyter_text>We can add the mutation function using the addMutationHandler of the pygenetic module which takes as parametersmutation_function = the mutation function to be usedweight = the weightage the mutation function needs to be given (mainly used when multiple mutations are added)<jupyter_code>ga.addMutationHandler(mutation,2,neuronsPerLayer,activations,optimizers)<jupyter_output><empty_output><jupyter_text>SelectionThe selection function `best` chooses the best (1 - cross_prob) percent of the population. Hence, this function is one of the possible selection handlers which can be used in our genetic algorithm<jupyter_code>ga.setSelectionHandler(Utils.SelectionHandlers.best)<jupyter_output><empty_output><jupyter_text>4. Time to Evolve This can be easily done using the `evolve` function of the GAEngine instance. It takes the `noOfIterations` as a parameter. Let's evolve it for 2 generations ( for demonstration purposes )<jupyter_code>ga.evolve(2)<jupyter_output>mutation_handlers_weights =  [1.0]
crossover_handlers_weights =  [1.0]
Epoch 1/30
768/768 [==============================] - 1s 805us/step - loss: 3.9776 - acc: 0.6510
Epoch 2/30
768/768 [==============================] - 0s 150us/step - loss: 2.9815 - acc: 0.6510
Epoch 3/30
768/768 [==============================] - 0s 252us/step - loss: 2.9265 - acc: 0.6510
Epoch 4/30
768/768 [==============================] - 0s 224us/step - loss: 2.5955 - acc: 0.6510
Epoch 5/30
768/768 [==============================] - 0s 151us/step - loss: 2.2763 - acc: 0.6510
Epoch 6/30
768/768 [==============================] - 0s 149us/step - loss: 2.1981 - acc: 0.6510
Epoch 7/30
768/768 [==============================] - 0s 144us/step - loss: 2.1357 - acc: 0.6510
Epoch 8/30
768/768 [==============================] - 0s 144us/step - loss: 2.0635 - acc: 0.6510
Epoch 9/30
768/768 [==============================] - 0s 182us/step - loss: 1.7269 - acc: 0.6510
Epoch 10/30
768/768 [==============================] - 0[...]<jupyter_text>We can get the best member by using the `best_fitness` attribute of the `GAEngine`. It returns a tuple having* chromsome having best fitness* best fitness value<jupyter_code>best = ga.best_fitness
print(best)<jupyter_output>([12, 'sigmoid', 10, 'relu', 'relu', 'adam'], 0.5369872798522314)<jupyter_text>5. Plotting the Statistics- The functionality for plotting the best, worst, average fitness values across iterations is present in `plot_statistics` function of statistics.py module. The function takes a list of attributes to be plotted.- These attributes can be `best-fitness`,`worst-fitness`,`avg-fitness`, `'diversity`, `mutation_rate`- The diversity and mutation rate values over iterations can also be visualized<jupyter_code>import matplotlib.pyplot as plt
fig = ga.statistics.plot_statistics(['best-fitness','worst-fitness','avg-fitness'])
plt.show()
fig = ga.statistics.plot_statistics(['diversity'])
plt.show()
fig = ga.statistics.plot_statistics(['mutation_rate'])
plt.show()<jupyter_output>best-fitness [0.5518418923020363, 0.5369872798522314]
worst-fitness [10.379614929358164, 10.37915732463201]
avg-fitness [2.8955850816642243, 3.0842435689022145]
{'fraction_non_alphanumeric': 0.12033389607953479, 'fraction_numerical': 0.04061151753892328, 'mean_word_length': 5.702074167190446, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '565824', 'license': 'MIT', 'n_tokens_mistral': 3609, 'n_tokens_neox': 3158, 'n_words': 1033, 'path': 'examples/ANN_topology/ANN-Topology.ipynb', 'repo_name': 'smithblack-0/pygenetic', 'size': '10662'}