<jupyter_start><jupyter_text>Testing the effect of *Dropout Rates* on Loss and Accuracy*This notebook uses the **Dog or Cat** data set available [here](http://files.fast.ai/data/dogscats.zip)*<jupyter_code># Put these at the top of every notebook, to get automatic reloading and inline plotting
%reload_ext autoreload
%autoreload 2
%matplotlib inline
# This file contains all the main external libs we'll use
from fastai.imports import *

from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *

# Using the same data set as Deep Learning Part 1 - lesson 1
PATH = "data/dogscats/"
# Select an image model
arch=resnet50
# Set to the number of CPU Cores you have
workers=8
# this is an array we will iterate through to test each one
dropouts = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]

# these variables are used for capturing results
los = []
acc = []<jupyter_output><empty_output><jupyter_text>Learning RateLet's see how learning rate is affected by batch size, starting with a small 64 batch size.<jupyter_code>sz=64 # 64x64 image size is very small and thus faster
bs=64 # batch size
learnrate = 5e-3 #0.005
tfms = tfms_from_model(arch, sz)
data = ImageClassifierData.from_paths(PATH, tfms=tfms, bs=bs, num_workers=workers)<jupyter_output><empty_output><jupyter_text>Let's try it with no dropouts (ps=0)<jupyter_code>learn = ConvLearner.pretrained(arch, data, precompute=False, ps=0)
# Finding the learning rate
lrf=learn.lr_find()
# Plotting the learning rate
learn.sched.plot()<jupyter_output><empty_output><jupyter_text>Not very smooth. Due to the small batch size, we need a very small learning rate. Now let's try it with 90% dropout (ps=0.9)<jupyter_code>learn = ConvLearner.pretrained(arch, data, precompute=False, ps=0.9)
# Finding the learning rate
lrf=learn.lr_find()
# Plotting the learning rate
learn.sched.plot()<jupyter_output><empty_output><jupyter_text>Time to change the batch size (increase by 8x) and see what happens<jupyter_code>sz=64 # unchanged
bs=512 # batch size
learnrate = 1e-1 #0.1
tfms = tfms_from_model(arch, sz)
data = ImageClassifierData.from_paths(PATH, tfms=tfms, bs=bs, num_workers=workers)<jupyter_output><empty_output><jupyter_text>Once again, start with no dropout.<jupyter_code>learn = ConvLearner.pretrained(arch, data, precompute=False, ps=0)
# Finding the learning rate
lrf=learn.lr_find()
# Plotting the learning rate
learn.sched.plot()<jupyter_output><empty_output><jupyter_text>Then switch to 0.9 dropout rate.<jupyter_code>learn = ConvLearner.pretrained(arch, data, precompute=False, ps=0.9)
# Finding the learning rate
lrf=learn.lr_find()
# Plotting the learning rate
learn.sched.plot()<jupyter_output><empty_output><jupyter_text>Learning Rate Conclusions:It's important to see that the best **learning rate** needs to be ***smaller*** when the **dropout rate** ***increases***. However, the effect is quite minor and playing it safe with a smaller learning rate goes a long way.  Start the Dropout TestsWe'll stick to the larger batch size and a learning rate of **0.1** for our tests.<jupyter_code>#loop through all entries in the dropouts array
for dropout in dropouts:
    learn = ConvLearner.pretrained(arch, data, precompute=False, ps=dropout)
    
    print('Dropout Rate:',dropout)
    # Make sure we have enough cycles and due to dropout variability, let's make sure we are mixing things up a bit
    %time fit_array = learn.fit(learnrate, 3, cycle_len=1, cycle_mult=2)
    
    # learn.fit actually returns an array that looks like this: [[0.1983260258436203],0.9185000009536743]
    # it includes that last val_loss and accuracy values
    accuracy = fit_array[1]
    loss = fit_array[0][0]
    # add them to our arrays for charting later
    acc.append(accuracy)
    los.append(loss)<jupyter_output>Dropout Rate: 0.0<jupyter_text>Looking at the *trn_loss* vs *val_loss* you can see it going from overfitting to underfitting. But if you went just by those values, you would be choosing a dropout rate of 0.3-0.5 which, as we'll see, isn't actually the best for our data set. Now that we have our results we print out the arrays so we don't have to rerun the entire notebook just to play with the charting<jupyter_code>print(dropouts)
print(acc)
print(los)<jupyter_output>[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
[0.9185000009536743, 0.9180000066757202, 0.9220000066757202, 0.9114999952316284, 0.9139999985694885, 0.9124999980926514, 0.907499997138977, 0.9019999947547913, 0.903999997138977, 0.8964999933242798]
[0.1983260258436203, 0.19329571437835694, 0.19230403864383697, 0.1986690136194229, 0.19804472887516023, 0.20221455514431, 0.20338530087471007, 0.209022167801857, 0.21730239295959472, 0.23208116745948792]<jupyter_text>Oh look, a convenient place to enter those arrays...<jupyter_code>#cheatsheet of results
#acc = [0.9185000009536743, 0.9180000066757202, 0.9220000066757202, 0.9114999952316284, 0.9139999985694885, 0.9124999980926514, 0.907499997138977, 0.9019999947547913, 0.903999997138977, 0.8964999933242798]
#los = [0.1983260258436203, 0.19329571437835694, 0.19230403864383697, 0.1986690136194229, 0.19804472887516023, 0.20221455514431, 0.20338530087471007, 0.209022167801857, 0.21730239295959472, 0.23208116745948792]<jupyter_output><empty_output><jupyter_text>Interpreting Results<jupyter_code>width = 0.05 # bar width
# Get current size
fig_size = plt.rcParams["figure.figsize"]
# Set new size (0=width,1=height)
fig_size[0] = 10
fig_size[1] = 6
plt.rcParams["figure.figsize"] = fig_size

plt.bar(dropouts, acc, width, align='center')

plt.tick_params(axis='x', which='major', labelsize=10)
plt.tick_params(axis='y', which='major', labelsize=15)

xrange = np.arange(0,1,.1)
plt.xticks(xrange, dropouts)
plt.ylim(.85,.95)

plt.xlabel('Dropout', size = 20)
plt.ylabel('Accuracy',size = 20)

plt.show()<jupyter_output><empty_output><jupyter_text>_**Note:** Larger is better._While accuracy doesn't vary greatly (2%), machine learning is all about small gains. A lot of small gains from hyperparameters add up to a really reliable model, not to mention a big difference in a Kaggle competition. Since we used **small images** for this run, we get clearer results from the **Accuracy** chart. The strongest case is 0.2 dropout rate with larger rates causing worse results.<jupyter_code>plt.bar(dropouts, los, width, align='center')

plt.tick_params(axis='x', which='major', labelsize=10)
plt.tick_params(axis='y', which='major', labelsize=15)

xrange = np.arange(0,1,.1)
plt.xticks(xrange, dropouts)
#plt.gca().invert_yaxis()

plt.xlabel('Dropout', size = 20)
plt.ylabel('Loss',size = 20)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10916641946010086, 'fraction_numerical': 0.1266686443191931, 'mean_word_length': 5.838742393509127, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '114923', 'license': 'MIT', 'n_tokens_mistral': 2952, 'n_tokens_neox': 2372, 'n_words': 692, 'path': 'Testing Dropout Rates (small images).ipynb', 'repo_name': 'ucffool/fastai-custom-learning-notebooks', 'size': '6742'}