<jupyter_start><jupyter_text>Setting Up The Model Before we get started with hyperparameter optimization, we need a model to optimize. Here, we'll be training a DenseNet201 on the CIFAR10 dataset.You won't get state of the art results, but that's intentional. We're not trying to create the best possible model, we're only going to evaluate different hyperparameter optimization schemes.<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>First, let's import the fastai library, a high level wrapper for PyTorch (similar to Keras for Tensorflow).This will help us quickly build a descent model so that we can focus on hyperparameter optimization.<jupyter_code>import fastai
from fastai.vision import *<jupyter_output><empty_output><jupyter_text>Now, we'll get a dataset and set up some basic data augmentation (which will be used later).Here's we're using a small subset of the MNIST dataset, so that the model trains really quickly and so that you get to see results in real time.But if you want to take it up a notch, replace `path = untar_data(URLs.MNIST_SAMPLE)` with `path = untar_data(URLs.CIFAR)`.You'll also want to increase the number of epochs to at least 3.<jupyter_code># Download the dataset
path = untar_data(URLs.MNIST_SAMPLE)

# Uncomment this line to use the CIFAR10 datast instead of MNIST
# path = untar_data(URLs.CIFAR)

# Set up the training and validation datasets
data = ImageDataBunch.from_folder(path, bs = 512).normalize()<jupyter_output><empty_output><jupyter_text>To make our life easier as we start to evaluate different hyperparameter optimization schemes, we'll define a `fit_with()` function that trains our model with certain hyperparameters.<jupyter_code># lr = learning rate
# wd = weight decay
# dp = dropout probability

def fit_with(lr,wd,dp):
    
    # Create the model using a specified dropout and weight decay
    learn = cnn_learner(data, models.resnet18, metrics=accuracy, ps=dp)
    
    
    # Train the model for a specified number of epochs using a specified max learning rate
    learn.fit_one_cycle(1, max_lr=lr)
    
    # Plot the loss over time
    learn.recorder.plot_losses()
    
    # Save, print, and return the model's accuracy
    acc = float(learn.validate(learn.data.valid_dl)[1])
    
    print("Current Accuracy:" + str(acc*100))
    
    return acc, learn<jupyter_output><empty_output><jupyter_text>Let's quickly train a basic model using random hyperparameters<jupyter_code>init_acc, init_model = fit_with(lr=1e-6, wd=0.4, dp=0.7)
init_model.save('/home/init_model')<jupyter_output>Accuracy:46.22178673744202<jupyter_text>Hmmm...That didn't turn out so great. Let's see if we can get something better by tuning the hyperparameters.  Grid Search For our first hyperparameter optimization algorithm, let's try grid search. Refer back to the [main article](https://blog.nanonets.com/hyp-opt) if you forgot how it works. Essentially, it's brute force. To execute Grid search, we need to first create a set of possible hyperparameter values.<jupyter_code>lrs = [1e-2, 1e-3, 1e-4]
wds = [1e-1, 1e-2, 1e-3]
dps = [0.1, 0.3, 0.5]<jupyter_output><empty_output><jupyter_text>Then we just loop over all the possible values, and save the model when the accuracy is the highest.In practice, grid search is usually paralellized, which each training run occuring on a different machine. However, the following code block uses `for` loops for simplicity and to run  in a simple notebook.**WARNING:** This code block can take an *extremely* long time to execute, so it is reccomended that you simply view the code.<jupyter_code>max_acc = 0

for lr in lrs:
  for wd in wds:
    for dp in dps:
      
      current_accuracy, model = fit_with(lr,wd,dp)
                
      if (current_accuracy > max_acc):
        model.save('/home/grid_search_best_model')
        max_acc = current_accuracy
        
        
print("The highest accuracy is: " + str(max_acc*100))<jupyter_output>Accuracy:61.82531714439392
The highest accuracy is: 99.36211705207825<jupyter_text>Random Search Random search is almost same as grid search, except we change the set of values that we loop over. Instead of a hard-coded set of values, we sample hyperparameters from a distribution i.e. choose them at random.<jupyter_code>import random

max_acc = 0
# The number of different hyperparameter values to try
num_iters = 10<jupyter_output><empty_output><jupyter_text>The second step is the same as that in grid search. Just try out the values and see what works best. **WARNING:** This code block can take an *extremely* long time to execute, so it is reccomended that you simply view the code.<jupyter_code>for iteration in range(num_iters):
  
  # Pick random hyperparameters in a specified range
  lr = 10**(-random.randint(3,7))
  wd = 10**(-random.randint(3,7))
  dp = random.uniform(0.1,0.5)
  
  current_accuracy, model = fit_with(lr,wd,dp)
  
  if (current_accuracy > max_acc):
    model.save('/home/grid_search_best_model')
    max_acc = current_accuracy
        
        
print("The highest accuracy is: " + str(max_acc*100))<jupyter_output>Accuracy:50.0490665435791
The highest accuracy is: 66.87929630279541<jupyter_text>Baysian Optimization This the moment you've all been waiting for.To implement Baysian optimization with a Gaussian process (explained in the [main article](https://blog.nanonets.com/hyp-opt)), we're going to use the `bayesian-optimization` [ package](https://github.com/fmfn/BayesianOptimization)<jupyter_code>!pip install bayesian-optimization<jupyter_output><empty_output><jupyter_text>Luckily, we already have a function to optimize: `fit_with()`!There's only one problem, the function won't work in it's current form with `baysian-optimization`.So we'll make one small change:  we're only going to return the accuracy, not the `learn` object.<jupyter_code>def fit_with(lr,wd,dp):
    
    # Create the model using a specified dropout and weight decay
    learn = cnn_learner(data, models.resnet18, metrics=accuracy, ps=dp)
    
    
    # Train the model for a specified number of epochs using a specified max learning rate
    learn.fit_one_cycle(1, max_lr=lr)
    
    # Plot the loss over time
    learn.recorder.plot_losses()
    
    # Save, print, and return the model's accuracy
    acc = float(learn.validate(learn.data.valid_dl)[1])
    
    print("Accuracy:" + str(acc*100))
    
    # The small change: we're only going to return the accuracy
    
    return acc<jupyter_output><empty_output><jupyter_text>Next, we define our Bayesian optimizer using the rules set by the `baysian-optimization` library, and let it do it's magic.**Note :** This step may also take a very long time (a few hours) to complete, but considerably less than random search and grid search.<jupyter_code>from bayes_opt import BayesianOptimization

# Bounded region of parameter space
pbounds = {'lr': (1e-4, 1e-2), 'wd':(1e-4,0.4), 'dp':(0.1,0.5)}

optimizer = BayesianOptimization(
    f=fit_with,
    pbounds=pbounds,
    verbose=2, # verbose = 1 prints only when a maximum is observed, verbose = 0 is silent
    random_state=1,
)

optimizer.maximize(init_points=2, n_iter=2,)



for i, res in enumerate(optimizer.res):
    print("Iteration {}: \n\t{}".format(i, res))

print(optimizer.max)<jupyter_output>Accuracy:59.273797273635864
| [0m 4       [0m | [0m 0.5927  [0m | [0m 0.5     [0m | [0m 0.0001  [0m | [0m 0.0001  [0m |
=============================================================
Iteration 0: 
	{'target': 0.9911677837371826, 'params': {'dp': 0.2668088018810296, 'lr': 0.007231212485077366, 'wd': 0.00014573848945622018}}
Iteration 1: 
	{'target': 0.9425907731056213, 'params': {'dp': 0.22093302905273593, 'lr': 0.0015528833190894193, 'wd': 0.037026204048042244}}
Iteration 2: 
	{'target': 0.9926398396492004, 'params': {'dp': 0.5, 'lr': 0.01, 'wd': 0.4}}
Iteration 3: 
	{'target': 0.5927379727363586, 'params': {'dp': 0.5, 'lr': 0.0001, 'wd': 0.0001}}
{'target': 0.9926398396492004, 'params': {'dp': 0.5, 'lr': 0.01, 'wd': 0.4}}
{'fraction_non_alphanumeric': 0.10660847880299251, 'fraction_numerical': 0.05511221945137157, 'mean_word_length': 4.593444909344491, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 47, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '168681', 'license': 'MIT', 'n_tokens_mistral': 2906, 'n_tokens_neox': 2563, 'n_words': 869, 'path': 'Hyperparameter_Optimization.ipynb', 'repo_name': 'ooluwaseunabel/Hyperparametrr-Optimisation-Techniques-Using-Python', 'size': '8020'}