<jupyter_start><jupyter_text>Cutting Cloud Costs with Infrastructure Automation (Part IV - Preparing and Training Code on AWS with Drop Out - MNIST)I said in the last post that I was ready to try this on our face. I'm totally not ready - I'm just missing one last thing I wanted to try. TFlearn gives us the option of easily including _**drop out**_ in our fully connected layers of the NN, and I wanted to explore that really quickly before moving on because I've read about it and it seems super easily to implement. Drop OutDrop out is a simple regularization concept in NNs. When we implement drop out, we're telling the NN to basically pretend like a few of the neurons in the fully connected layer don't exist. This forces the other neurons (who have not been dropped out) to remodel their weights / thresholds to work with different neurons in every iteration of training.![](http://cs231n.github.io/assets/nn2/dropout.jpeg)I like to think of this method employing the same logic that subsampling does in tree techniques like random forests. When we subsample in random forest, we choose only a subset of samples and / or features to train on. This basically forces different features to explicitly coexist with some features and prevent the coexistence with other features to get a bit more perspective on which features are truly more important.Dropping out within NNs also force features (or linear combinations of features, depending on how deep we get into our fully connected layers) to work nice with other features to see which really end up making an impact in every situation before completing our model training.Dropping out in TFlearn is super simple. There's literally a _**dropout**_ function where you tell it what percentage of the neurons to keep in every iteration and TFlearn will perform this for you by dropping our random neurons.It's actually been in the code this whole time because sentdex used dropout in his model and I just copied sentdex, but I've been commenting it out. All the code below is _**the exact same as the previous post**_ except the dropout line when building our NN.<jupyter_code># Install tflearn when working on a fresh AWS Deep Learning AMI (which doesn't come with TFlearn)
import os
os.system("sudo pip install tflearn")
# TFlearn libraries
import tflearn
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.estimator import regression
import tflearn.datasets.mnist as mnist

# General purpose libraries
import matplotlib.pyplot as plt
import numpy as np
import math
# Extract data from mnist.load_data()
x, y, x_test, y_test = mnist.load_data(one_hot = True)
# Reshape x
x_reshaped = x.reshape([-1, 28, 28, 1])
print 'x_reshaped has the shape {}'.format(x_reshaped.shape)
# Reshape x_test
x_test_reshaped = x_test.reshape([-1, 28, 28, 1])
print 'x_test_reshaped has the shape {}'.format(x_test_reshaped.shape)
# sentdex's code to build the neural net using tflearn
#   Input layer --> conv layer w/ max pooling --> conv layer w/ max pooling --> fully connected layer --> output layer
convnet = input_data(shape = [None, 28, 28, 1], name = 'input')

convnet = conv_2d(convnet, 32, 2, activation = 'relu')
convnet = max_pool_2d(convnet, 2)

convnet = conv_2d(convnet, 64, 2, activation = 'relu')
convnet = max_pool_2d(convnet, 2)

convnet = fully_connected(convnet, 1024, activation = 'relu')
convnet = dropout(convnet, 0.8)

convnet = fully_connected(convnet, 10, activation = 'softmax')
convnet = regression(convnet, optimizer = 'sgd', learning_rate = 0.01, loss = 'categorical_crossentropy', name = 'targets')
model = tflearn.DNN(convnet)
model.fit(
    {'input': x_reshaped}, 
    {'targets': y}, 
    n_epoch = 5, 
    validation_set = ({'input': x_test_reshaped}, {'targets': y_test}), 
    snapshot_step = 500, 
    show_metric = True
)<jupyter_output>Training Step: 4299  | total loss: [1m[32m0.11463[0m[0m | time: 11.579s
| SGD | epoch: 005 | loss: 0.11463 - acc: 0.9676 -- iter: 54976/55000
Training Step: 4300  | total loss: [1m[32m0.11163[0m[0m | time: 12.897s
| SGD | epoch: 005 | loss: 0.11163 - acc: 0.9693 | val_loss: 0.09128 - val_acc: 0.9733 -- iter: 55000/55000
--
{'fraction_non_alphanumeric': 0.08097261567516525, 'fraction_numerical': 0.03517469310670444, 'mean_word_length': 4.634308510638298, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '351403', 'license': 'Apache-2.0', 'n_tokens_mistral': 1382, 'n_tokens_neox': 1226, 'n_words': 571, 'path': 'notebook/9 - Cutting Cloud Costs with Infrastructure Automation (Part IV - Preparing and Training Code on AWS with Drop Out - MNIST).ipynb', 'repo_name': 'strikingmoose/chi_lars_face_detection', 'size': '4236'}