<jupyter_start><jupyter_text>Optimization of the snake parameters The goal is to find the best parametrization for Cobra, i.e get the parameters which make the learninship as efficient as possible. First let import the files we need :<jupyter_code>import os
import sys
import math
import pygame
import random
import time

import tensorflow
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import Adam

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

from collections import deque

import numpy as np

from Env import Env
from Viper import Viper
from Cobra import Cobra
from main import cobra_main<jupyter_output><empty_output><jupyter_text>To get the best parameters, you will train different Cobras using different values. The parameters we are going to modify are the followings :- self.epsilon_decay- self.learning_rate- self.batch_size  Optimization of Learning Rate and  the Epsilon Decay First, we are looking for the best couple (epsilon_decay, learning_rate) :<jupyter_code>l_epsilon_rate = np.linspace(0.75,0.999,5)
l_rate = np.linspace(0.0005,0.0015,5)
print(l_epsilon_rate)
print(l_rate)

matrix_param = np.array([l_epsilon_rate, l_rate]).T
print(matrix_param)
matrix_cobra = []

for eps in l_epsilon_rate :
    cobra_line = []
    for rate in l_rate :
        cobra_line.append(Cobra(epsilon_decay = eps, learning_rate = rate))
    matrix_cobra.append(cobra_line)
print('done')<jupyter_output>done<jupyter_text>**matrix_cobra** now contains a 2D-array composed by cobra using different parameters :<jupyter_code>matrix_cobra = np.array(matrix_cobra)

print(matrix_cobra)
print(matrix_cobra.shape)<jupyter_output>[[<__main__.Cobra object at 0x7fb0a0ada690>
  <__main__.Cobra object at 0x7fb08557ded0>
  <__main__.Cobra object at 0x7fb086be19d0>
  <__main__.Cobra object at 0x7fb0a3cc7dd0>
  <__main__.Cobra object at 0x7fb0a0866d90>]
 [<__main__.Cobra object at 0x7fb0a4247f50>
  <__main__.Cobra object at 0x7fb087e32890>
  <__main__.Cobra object at 0x7fb0857b4f10>
  <__main__.Cobra object at 0x7fb0a0a0d990>
  <__main__.Cobra object at 0x7fb087e4eb50>]
 [<__main__.Cobra object at 0x7fb087ed43d0>
  <__main__.Cobra object at 0x7fb087e89810>
  <__main__.Cobra object at 0x7fb0a0ada7d0>
  <__main__.Cobra object at 0x7fb087260e90>
  <__main__.Cobra object at 0x7fb085472450>]
 [<__main__.Cobra object at 0x7fb0a3f9ba10>
  <__main__.Cobra object at 0x7fb0a2df8410>
  <__main__.Cobra object at 0x7fb0877e60d0>
  <__main__.Cobra object at 0x7fb0a31d4650>
  <__main__.Cobra object at 0x7fb087c35750>]
 [<__main__.Cobra object at 0x7fb0a2d80490>
  <__main__.Cobra object at 0x7fb0a44ba690>
  <__main__.Cobra object at [...]<jupyter_text>Cobra's Training part  In the following cell, each **cobra** from **matrix_cobra** will have a training by using the **cobra_main** contained in **main.py**. After each training, a mean of ten games'score will be saved in **l_score**. Note that the training is composed by 200 generations with 500 maximum actions for each of them.<jupyter_code>l_score = []
evol_score = []

for cobra_line in matrix_cobra :
    score_line = []
    score_evol_line = []
    for cobra in cobra_line  :
        
        print("epsilon_decay : ", cobra.epsilon_decay)
        print("learning_rate : ", cobra.learning_rate)
        
        score_evol, real_score = cobra_main(dqn_agent = cobra, nb_gens = 200, max_action = 500, show = False)
        
        score_line.append(np.mean(real_score))
        score_evol_line.append(score_evol)
        print('The score :', np.mean(real_score))
        time.sleep(2)
    l_score.append(score_line)
    evol_score.append(score_evol_line)
    

print(l_score)<jupyter_output>epsilon_decay :  0.75
learning_rate :  0.0005
Game :  0  | Epsilon is  1.0
Game :  1  | Epsilon is  0.75
Game :  2  | Epsilon is  0.5625
Game :  3  | Epsilon is  0.421875
Game :  4  | Epsilon is  0.31640625
Game :  5  | Epsilon is  0.2373046875
Game :  6  | Epsilon is  0.177978515625
Game :  7  | Epsilon is  0.13348388671875
Game :  8  | Epsilon is  0.1001129150390625
Game :  9  | Epsilon is  0.07508468627929688
Game :  10  | Epsilon is  0.056313514709472656
Game :  11  | Epsilon is  0.04223513603210449
Game :  12  | Epsilon is  0.03167635202407837
Game :  13  | Epsilon is  0.023757264018058777
Game :  14  | Epsilon is  0.017817948013544083
Game :  15  | Epsilon is  0.017817948013544083
Game :  16  | Epsilon is  0.013363461010158062
Game :  17  | Epsilon is  0.010022595757618546
Game :  18  | Epsilon is  0.01
Game :  19  | Epsilon is  0.01
Game :  20  | Epsilon is  0.01
Game :  21  | Epsilon is  0.01
Game :  22  | Epsilon is  0.01
Game :  23  | Epsilon is  0.01
Game :  24  | Epsilon is[...]<jupyter_text>The score is evaluated on **ten games**. It is now contained in an array. Each line refers to an epsilon_decay, each column refers to a learning_rate :<jupyter_code>l_score = np.array(l_score)
print(l_score)

#save_score = [[1.0, 13.3, 0.6, 9.3, 0.3], [5.9, 9.4, 0.6, 0.0, 13.4], [0.0, 10.8, 0.6, 13.4, 0.1], [2.0, 3.3, 1.3, 18.3, 9.3], [0.1, 0.3, 0.2, 0.3, 0.3]]<jupyter_output>[[ 1.  13.3  0.6  9.3  0.3]
 [ 5.9  9.4  0.6  0.  13.4]
 [ 0.  10.8  0.6 13.4  0.1]
 [ 2.   3.3  1.3 18.3  9.3]
 [ 0.1  0.3  0.2  0.3  0.3]]<jupyter_text>3D-Representation of the results  !!! Warning : The following line needs to be run once only !!!<jupyter_code>l_epsilon_rate, l_rate = np.meshgrid(l_epsilon_rate,l_rate)
print(l_epsilon_rate)<jupyter_output>[[0.75    0.81225 0.8745  0.93675 0.999  ]
 [0.75    0.81225 0.8745  0.93675 0.999  ]
 [0.75    0.81225 0.8745  0.93675 0.999  ]
 [0.75    0.81225 0.8745  0.93675 0.999  ]
 [0.75    0.81225 0.8745  0.93675 0.999  ]]<jupyter_text>----------------------------------------------------------- The following cell provides a 3D-representation of the score :<jupyter_code>%matplotlib

graph = plt.axes(projection='3d', xlabel = 'Epsilon_decay', ylabel = 'Learnin_rate')
graph.plot_surface(l_epsilon_rate, l_rate, l_score, cmap = 'plasma') 
plt.title('The score')<jupyter_output>Using matplotlib backend: MacOSX<jupyter_text>Get the best parametrization<jupyter_code>#Cell to delete the meshgrid effect 
l_epsilon_rate = np.linspace(0.75,0.999,5)
l_rate = np.linspace(0.0005,0.0015,5)<jupyter_output><empty_output><jupyter_text>We want to get the best parametrization values which can be severals. In order to do that, let use **np.where(l_score == l_score.max())**. It returns a tuple containing the coordonates (length = 2). The problem is that the first element contains the line and the second, the column. The following code provides a solution to switch it into an array with the form : [param1[line, column], param2[column], .. etc ..]<jupyter_code>best_parametrization_index = np.where(l_score == l_score.max()) #Il peut y avoir plusieurs max 
print('The best score is : ', l_score.max())
print(np.where(l_score == l_score.max())) 

#In order to use zip, we need to retype the arrays into lists
array0 = list(best_parametrization_index[0])
print('Index of the best epsilon :', array0)
array1 = list(best_parametrization_index[1])
print('Index of the best learning rate :', array1)

best_parametrization = []
for eps_index, learning_index in zip(array0,array1):
    best_parametrization.append((l_epsilon_rate[eps_index], l_rate[learning_index]))
print('List of best parametrization:', best_parametrization)
    
#best_parametrization[0][i] for i in {0,1} are respectively the epsilon_decay and the learning_rate which provide the most efficient learning<jupyter_output>The best score is :  18.3
(array([3]), array([3]))
Index of the best epsilon : [3]
Index of the best learning rate : [3]
List of best parametrization: [(0.93675, 0.00125)]<jupyter_text>The best parametrization seems to be **Epsilon Decay = 0.93675, Learning Rate = 0.00125**. In this case, the best parameters are unique, however they could have been severals.  Optimization of the Batch Size The best parametrizations are now contained in **best_parametrization**. These value will provide the most efficient learning for Cobra. Now we want to find the best **batch_size**. We will use the same process. First let creates a cobra_matrix :<jupyter_code>matrix_cobra_bis = [] #As the previous matrix : it will contains cobras using different init parameters

l_batch_size = np.linspace(30,250,11)
print(l_batch_size)

for param in best_parametrization :
    cobra_line_bis = []
    for batch_sz in l_batch_size :
        cobra_line_bis.append(Cobra(epsilon_decay = param[0], learning_rate = param[1], batch_size = batch_sz))
    matrix_cobra_bis.append(cobra_line_bis)

matrix_cobra_bis = np.array(matrix_cobra_bis)
print(matrix_cobra_bis)<jupyter_output>[ 30.  52.  74.  96. 118. 140. 162. 184. 206. 228. 250.]
[[<Cobra.Cobra object at 0x7fd072d0ded0>
  <Cobra.Cobra object at 0x7fd073049390>
  <Cobra.Cobra object at 0x7fd073095410>
  <Cobra.Cobra object at 0x7fd0730562d0>
  <Cobra.Cobra object at 0x7fd073110dd0>
  <Cobra.Cobra object at 0x7fd073155b50>
  <Cobra.Cobra object at 0x7fd07312a750>
  <Cobra.Cobra object at 0x7fd07318ec10>
  <Cobra.Cobra object at 0x7fd0731d7dd0>
  <Cobra.Cobra object at 0x7fd073154690>
  <Cobra.Cobra object at 0x7fd0731bfc90>]]<jupyter_text>Then, each Cobra get a training of **200 generations** :<jupyter_code>batch_score = []
batch_score_progress = []


for cobra_line_bis in matrix_cobra_bis :
    batch_score_line = []
    batch_score_progress_line = []
    for cobra in cobra_line_bis  :
        
        print("epsilon_decay : ", cobra.epsilon_decay)
        print("learning_rate : ", cobra.learning_rate)
        print("batch size : ", cobra.batch_size)
        
        score_evol, real_score = cobra_main(dqn_agent = cobra, nb_gens = 200, max_action = 500, show = False)
        
        batch_score_line.append(np.mean(real_score))
        batch_score_progress_line.append(score_evol)
        print('The score :', np.mean(real_score))
    batch_score.append(batch_score_line)
    batch_score_progress.append(batch_score_progress_line)
    

print(batch_score)
#Let save the batch score in the following array : 
batch_score = [[1.0, 2.0, 0.0, 14.0, 2.0, 1.0, 0.0, 0.0, 8.0, 10.0, 0.0]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10504694835680752, 'fraction_numerical': 0.09839593114241002, 'mean_word_length': 4.122745490981964, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '283488', 'license': 'MIT', 'n_tokens_mistral': 4442, 'n_tokens_neox': 3904, 'n_words': 1030, 'path': 'Notebook.ipynb', 'repo_name': 'alexandrefresnais/Cobra', 'size': '10224'}