<jupyter_start><jupyter_text>Deep Learning=============Assignment 3 and Solutions------------Previously in [2_fullyconnected.ipynb](./2_fullyconnected.ipynb), you trained a logistic regression and a neural network model.The goal of this assignment is to explore regularization techniques.<jupyter_code># These are all the modules we'll be using later. Make sure you can import them
# before proceeding further.
from __future__ import print_function
import numpy as np
import tensorflow as tf
from six.moves import cPickle as pickle<jupyter_output><empty_output><jupyter_text>First reload the data we generated in _notmist.ipynb_.<jupyter_code>pickle_file = 'notMNIST.pickle'

with open(pickle_file, 'rb') as f:
    save = pickle.load(f)
    train_dataset = save['train_dataset']
    train_labels = save['train_labels']
    valid_dataset = save['valid_dataset']
    valid_labels = save['valid_labels']
    test_dataset = save['test_dataset']
    test_labels = save['test_labels']
    del save  # Hint to help gc free up memory
    print('Training set', train_dataset.shape, train_labels.shape)
    print('Validation set', valid_dataset.shape, valid_labels.shape)
    print('Test set', test_dataset.shape, test_labels.shape)<jupyter_output>Training set (200000, 28, 28) (200000,)
Validation set (10000, 28, 28) (10000,)
Test set (10000, 28, 28) (10000,)<jupyter_text>Reformat into a shape that's more adapted to the models we're going to train:- data as a flat matrix,- labels as float 1-hot encodings.<jupyter_code>IMAGE_SIZE = 28
NUM_LABELS = 10

def reformat(dataset, labels):
    dataset = dataset.reshape((-1, IMAGE_SIZE * IMAGE_SIZE)).astype(np.float32)
    # Map 2 to [0.0, 1.0, 0.0 ...], 3 to [0.0, 0.0, 1.0 ...]
    labels = (np.arange(NUM_LABELS) == labels[:,None]).astype(np.float32)
    return dataset, labels

train_dataset, train_labels = reformat(train_dataset, train_labels)
valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)
test_dataset, test_labels = reformat(test_dataset, test_labels)
print('Training set', train_dataset.shape, train_labels.shape)
print('Validation set', valid_dataset.shape, valid_labels.shape)
print('Test set', test_dataset.shape, test_labels.shape)
def accuracy(predictions, labels):
    return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
            / predictions.shape[0])<jupyter_output><empty_output><jupyter_text>---Problem 1---------Introduce and tune L2 regularization for both logistic and neural network models. Remember that L2 amounts to adding a penalty on the norm of the weights to the loss. In TensorFlow, you can compute the L2 loss for a tensor `t` using `nn.l2_loss(t)`. The right amount of regularization should improve your validation / test accuracy.<jupyter_code># Logistic regression with L2 regulariation
NUM_LABELS = 10
IMAGE_SIZE = 28
batch_size = 128
num_steps = 801
beta = 0.001
graph = tf.Graph()
with graph.as_default():
    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32, 
                                      shape=(batch_size, IMAGE_SIZE * IMAGE_SIZE))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, NUM_LABELS))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)
  
    # Variables.
    weights_output = tf.Variable(
        tf.truncated_normal([IMAGE_SIZE * IMAGE_SIZE, NUM_LABELS]))
    biases_output = tf.Variable(tf.zeros([NUM_LABELS]))
  
    # Training computation.
    score = tf.matmul(tf_train_dataset, weights_output) + biases_output
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(score, tf_train_labels))\
        + beta * tf.nn.l2_loss(weights_output)
  
    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)
  
    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(score)
    score_valid = tf.matmul(tf_valid_dataset, weights_output) + biases_output
    valid_prediction = tf.nn.softmax(score_valid)
    score_test = tf.matmul(tf_test_dataset, weights_output) + biases_output
    test_prediction = tf.nn.softmax(score_test)
with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
        _, l, predictions = session.run(
            [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 100 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
                  valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))
# Simple neural network with L2 regulariation
NUM_LABELS = 10
IMAGE_SIZE = 28
batch_size = 128
num_hidden1 = 1024
num_steps = 10001
beta = 0.001
graph = tf.Graph()
with graph.as_default():
    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32, 
                                      shape=(batch_size, IMAGE_SIZE * IMAGE_SIZE))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, NUM_LABELS))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)
  
    # Variables.
    weights_hidden1 = tf.Variable(
        tf.truncated_normal([IMAGE_SIZE * IMAGE_SIZE, num_hidden1]))
    biases_hidden1 = tf.Variable(tf.zeros([num_hidden1]))
    weights_output = tf.Variable(
        tf.truncated_normal([num_hidden1, NUM_LABELS]))
    biases_output = tf.Variable(tf.zeros([NUM_LABELS]))
  
    # Training computation.
    score_hidden = tf.matmul(tf_train_dataset, weights_hidden1) + biases_hidden1
    hidden1 = tf.nn.relu(score_hidden)
    
    score = tf.matmul(hidden1, weights_output) + biases_output
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(score, tf_train_labels))\
        + beta * (tf.nn.l2_loss(weights_hidden1) + tf.nn.l2_loss(weights_output))
  
    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)
  
    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(score)
    score_hidden_valid = tf.matmul(tf_valid_dataset, weights_hidden1) + biases_hidden1
    score_valid = tf.matmul(
        tf.nn.relu(score_hidden_valid), weights_output) + biases_output
    valid_prediction = tf.nn.softmax(score_valid)
    score_hidden_test = tf.matmul(tf_test_dataset, weights_hidden1) + biases_hidden1
    score_test = tf.matmul(
        tf.nn.relu(score_hidden_test), weights_output) + biases_output
    test_prediction = tf.nn.softmax(score_test)
with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
        _, l, predictions = session.run(
            [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
                  valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))<jupyter_output>Initialized
Minibatch loss at step 0: 580.127075
Minibatch accuracy: 11.7%
Validation accuracy: 25.9%
Minibatch loss at step 500: 197.604950
Minibatch accuracy: 81.2%
Validation accuracy: 78.7%
Minibatch loss at step 1000: 115.851494
Minibatch accuracy: 77.3%
Validation accuracy: 81.6%
Minibatch loss at step 1500: 68.860794
Minibatch accuracy: 87.5%
Validation accuracy: 82.7%
Minibatch loss at step 2000: 41.223942
Minibatch accuracy: 89.8%
Validation accuracy: 84.8%
Minibatch loss at step 2500: 25.145836
Minibatch accuracy: 91.4%
Validation accuracy: 85.3%
Minibatch loss at step 3000: 15.510725
Minibatch accuracy: 85.9%
Validation accuracy: 86.2%
Minibatch loss at step 3500: 9.636346
Minibatch accuracy: 85.2%
Validation accuracy: 87.2%
Minibatch loss at step 4000: 5.957050
Minibatch accuracy: 90.6%
Validation accuracy: 87.7%
Minibatch loss at step 4500: 3.780931
Minibatch accuracy: 89.1%
Validation accuracy: 87.1%
Minibatch loss at step 5000: 2.511935
Minibatch accuracy: 89.8%
Validati[...]<jupyter_text>---Problem 2---------Let's demonstrate an extreme case of overfitting. Restrict your training data to just a few batches. What happens?<jupyter_code># Logistic regression with L2 regulariation under overfitting
# by restricting training data to just 3 batches.
NUM_LABELS = 10
IMAGE_SIZE = 28
batch_size = 128
num_steps = 801
beta = 0.001
graph = tf.Graph()
with graph.as_default():
    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32, 
                                      shape=(batch_size, IMAGE_SIZE * IMAGE_SIZE))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, NUM_LABELS))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)
  
    # Variables.
    weights_output = tf.Variable(
        tf.truncated_normal([IMAGE_SIZE * IMAGE_SIZE, NUM_LABELS]))
    biases_output = tf.Variable(tf.zeros([NUM_LABELS]))
  
    # Training computation.
    score = tf.matmul(tf_train_dataset, weights_output) + biases_output
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(score, tf_train_labels))\
        + beta * tf.nn.l2_loss(weights_output)
  
    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)
  
    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(score)
    score_valid = tf.matmul(tf_valid_dataset, weights_output) + biases_output
    valid_prediction = tf.nn.softmax(score_valid)
    score_test = tf.matmul(tf_test_dataset, weights_output) + biases_output
    test_prediction = tf.nn.softmax(score_test)
with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        # offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        offset = (step % 3) * batch_size
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
        _, l, predictions = session.run(
            [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 100 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
                  valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))
# Simple neural network with L2 regulariation under overfitting
# by restricting training data to just 3 batches.
NUM_LABELS = 10
IMAGE_SIZE = 28
batch_size = 128
num_hidden1 = 1024
num_steps = 10001
beta = 0.001
graph = tf.Graph()
with graph.as_default():
    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32, 
                                      shape=(batch_size, IMAGE_SIZE * IMAGE_SIZE))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, NUM_LABELS))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)
  
    # Variables.
    weights_hidden1 = tf.Variable(
        tf.truncated_normal([IMAGE_SIZE * IMAGE_SIZE, num_hidden1]))
    biases_hidden1 = tf.Variable(tf.zeros([num_hidden1]))
    weights_output = tf.Variable(
        tf.truncated_normal([num_hidden1, NUM_LABELS]))
    biases_output = tf.Variable(tf.zeros([NUM_LABELS]))
  
    # Training computation.
    score_hidden = tf.matmul(tf_train_dataset, weights_hidden1) + biases_hidden1
    hidden1 = tf.nn.relu(score_hidden)
    
    score = tf.matmul(hidden1, weights_output) + biases_output
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(score, tf_train_labels))\
        + beta * (tf.nn.l2_loss(weights_hidden1) + tf.nn.l2_loss(weights_output))
  
    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)
  
    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(score)
    score_hidden_valid = tf.matmul(tf_valid_dataset, weights_hidden1) + biases_hidden1
    score_valid = tf.matmul(
        tf.nn.relu(score_hidden_valid), weights_output) + biases_output
    valid_prediction = tf.nn.softmax(score_valid)
    score_hidden_test = tf.matmul(tf_test_dataset, weights_hidden1) + biases_hidden1
    score_test = tf.matmul(
        tf.nn.relu(score_hidden_test), weights_output) + biases_output
    test_prediction = tf.nn.softmax(score_test)
with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        # offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        offset = (step % 3) * batch_size
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
        _, l, predictions = session.run(
            [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
                  valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))<jupyter_output>Initialized
Minibatch loss at step 0: 632.554932
Minibatch accuracy: 7.0%
Validation accuracy: 30.1%
Minibatch loss at step 500: 190.373688
Minibatch accuracy: 100.0%
Validation accuracy: 75.7%
Minibatch loss at step 1000: 115.453049
Minibatch accuracy: 100.0%
Validation accuracy: 75.7%
Minibatch loss at step 1500: 70.017151
Minibatch accuracy: 100.0%
Validation accuracy: 75.7%
Minibatch loss at step 2000: 42.462204
Minibatch accuracy: 100.0%
Validation accuracy: 75.8%
Minibatch loss at step 2500: 25.751610
Minibatch accuracy: 100.0%
Validation accuracy: 75.7%
Minibatch loss at step 3000: 15.617757
Minibatch accuracy: 100.0%
Validation accuracy: 75.7%
Minibatch loss at step 3500: 9.473000
Minibatch accuracy: 100.0%
Validation accuracy: 75.9%
Minibatch loss at step 4000: 5.749737
Minibatch accuracy: 100.0%
Validation accuracy: 76.1%
Minibatch loss at step 4500: 3.497044
Minibatch accuracy: 100.0%
Validation accuracy: 76.9%
Minibatch loss at step 5000: 2.133898
Minibatch accuracy: 100.0%[...]<jupyter_text>---Problem 3---------Introduce Dropout on the hidden layer of the neural network. Remember: Dropout should only be introduced during training, not evaluation, otherwise your evaluation results would be stochastic as well. TensorFlow provides `nn.dropout()` for that, but you have to make sure it's only inserted during training.What happens to our extreme overfitting case?<jupyter_code># Simple neural network with dropout
NUM_LABELS = 10
IMAGE_SIZE = 28
batch_size = 128
num_hidden1 = 1024
num_steps = 10001
beta = 0.001
keep_prob = 0.3
graph = tf.Graph()
with graph.as_default():
    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32, 
                                      shape=(batch_size, IMAGE_SIZE * IMAGE_SIZE))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, NUM_LABELS))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)
  
    # Variables.
    weights_hidden1 = tf.Variable(
        tf.truncated_normal([IMAGE_SIZE * IMAGE_SIZE, num_hidden1]))
    biases_hidden1 = tf.Variable(tf.zeros([num_hidden1]))
    weights_output = tf.Variable(
        tf.truncated_normal([num_hidden1, NUM_LABELS]))
    biases_output = tf.Variable(tf.zeros([NUM_LABELS]))
  
    # Training computation.
    score_hidden = tf.matmul(tf_train_dataset, weights_hidden1) + biases_hidden1
    hidden1 = tf.nn.relu(score_hidden)
    # Add dropout
    hidden1_drop = tf.nn.dropout(hidden1, keep_prob)
    
    score = tf.matmul(hidden1_drop, weights_output) + biases_output
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(score, tf_train_labels))
  
    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)
  
    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(score)
    score_hidden_valid = tf.matmul(tf_valid_dataset, weights_hidden1) + biases_hidden1
    score_valid = tf.matmul(
        tf.nn.relu(score_hidden_valid), weights_output) + biases_output
    valid_prediction = tf.nn.softmax(score_valid)
    score_hidden_test = tf.matmul(tf_test_dataset, weights_hidden1) + biases_hidden1
    score_test = tf.matmul(
        tf.nn.relu(score_hidden_test), weights_output) + biases_output
    test_prediction = tf.nn.softmax(score_test)
with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {tf_train_dataset: batch_data, tf_train_labels: batch_labels}
        _, l, predictions = session.run(
            [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
                  valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))
# Simple neural network with dropout under overfitting
# by restricting training data to just 3 batches.
NUM_LABELS = 10
IMAGE_SIZE = 28
batch_size = 128
num_hidden1 = 1024
num_steps = 10001
beta = 0.001
graph = tf.Graph()
with graph.as_default():
    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32, 
                                      shape=(batch_size, IMAGE_SIZE * IMAGE_SIZE))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, NUM_LABELS))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)
  
    # Variables.
    weights_hidden1 = tf.Variable(
        tf.truncated_normal([IMAGE_SIZE * IMAGE_SIZE, num_hidden1]))
    biases_hidden1 = tf.Variable(tf.zeros([num_hidden1]))
    weights_output = tf.Variable(
        tf.truncated_normal([num_hidden1, NUM_LABELS]))
    biases_output = tf.Variable(tf.zeros([NUM_LABELS]))
  
    # Training computation.
    score_hidden = tf.matmul(tf_train_dataset, weights_hidden1) + biases_hidden1
    hidden1 = tf.nn.relu(score_hidden)
    # Add dropout
    hidden1_drop = tf.nn.dropout(hidden1, keep_prob)
    
    score = tf.matmul(hidden1_drop, weights_output) + biases_output
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(score, tf_train_labels))
  
    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)
  
    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(score)
    score_hidden_valid = tf.matmul(tf_valid_dataset, weights_hidden1) + biases_hidden1
    score_valid = tf.matmul(
        tf.nn.relu(score_hidden_valid), weights_output) + biases_output
    valid_prediction = tf.nn.softmax(score_valid)
    score_hidden_test = tf.matmul(tf_test_dataset, weights_hidden1) + biases_hidden1
    score_test = tf.matmul(
        tf.nn.relu(score_hidden_test), weights_output) + biases_output
    test_prediction = tf.nn.softmax(score_test)
with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        # offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        offset = (step % 3) * batch_size
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
        _, l, predictions = session.run(
            [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
                  valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))<jupyter_output>Initialized
Minibatch loss at step 0: 602.876953
Minibatch accuracy: 10.9%
Validation accuracy: 34.4%
Minibatch loss at step 500: 0.176566
Minibatch accuracy: 99.2%
Validation accuracy: 77.0%
Minibatch loss at step 1000: 1.866451
Minibatch accuracy: 97.7%
Validation accuracy: 77.9%
Minibatch loss at step 1500: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 78.2%
Minibatch loss at step 2000: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 77.9%
Minibatch loss at step 2500: 0.083202
Minibatch accuracy: 99.2%
Validation accuracy: 77.9%
Minibatch loss at step 3000: 0.153027
Minibatch accuracy: 99.2%
Validation accuracy: 77.6%
Minibatch loss at step 3500: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 77.9%
Minibatch loss at step 4000: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 78.1%
Minibatch loss at step 4500: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 78.2%
Minibatch loss at step 5000: 0.000000
Minibatch accuracy: 100.0%
Validation[...]<jupyter_text>---Problem 4---------Try to get the best performance you can using a multi-layer model! The best reported test accuracy using a deep network is [97.1%](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html?showComment=1391023266211c8758720086795711595).One avenue you can explore is to add multiple layers.Another one is to use learning rate decay:    global_step = tf.Variable(0)   count the number of steps taken.    learning_rate = tf.train.exponential_decay(0.5, global_step, ...)    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)<jupyter_code># Deep learning with 4-layers neural networks (3 hidden layers + 1 output layer), 
# dropout, L2 regularization and learning rate decay.
NUM_LABELS = 10
IMAGE_SIZE = 28
batch_size = 128
num_hidden1 = 1024
num_hidden2 = 600
num_hidden3 = 200
sd_hidden1 = np.sqrt(2.0/1000)
sd_hidden2 = np.sqrt(2.0/num_hidden1)
sd_hidden3 = np.sqrt(2.0/num_hidden2)
sd_output = np.sqrt(2.0/num_hidden3)
keep_prob1 = 0.95
keep_prob2 = 0.6
keep_prob3 = 0.7
beta1 = 0.0001
beta2 = 0.0001
beta3 = 0.0001
beta4 = 0.0001
num_steps = 20001
start_learning_rate = 0.4
decay_steps = 1000
decay_rate = 0.96
graph = tf.Graph()
with graph.as_default():
    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32, 
                                      shape=(batch_size, IMAGE_SIZE * IMAGE_SIZE))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, NUM_LABELS))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)
  
    # Variables.
    weights_hidden1 = tf.Variable(
        tf.truncated_normal([IMAGE_SIZE * IMAGE_SIZE, num_hidden1], stddev=sd_hidden1))
    biases_hidden1 = tf.Variable(tf.zeros([num_hidden1]))
    weights_hidden2 = tf.Variable(
        tf.truncated_normal([num_hidden1, num_hidden2], stddev=sd_hidden2))
    biases_hidden2 = tf.Variable(tf.zeros([num_hidden2]))
    weights_hidden3 = tf.Variable(
        tf.truncated_normal([num_hidden2, num_hidden3], stddev=sd_hidden3))
    biases_hidden3 = tf.Variable(tf.zeros([num_hidden3]))
    weights_output = tf.Variable(
        tf.truncated_normal([num_hidden3, NUM_LABELS], stddev=sd_output))
    biases_output = tf.Variable(tf.zeros([NUM_LABELS]))
  
    # Computate training.
    score_hidden1 = tf.matmul(tf_train_dataset, weights_hidden1) + biases_hidden1
    hidden1 = tf.nn.relu(score_hidden1)
    hidden1_drop = tf.nn.dropout(hidden1, keep_prob1)
    score_hidden2 = tf.matmul(hidden1_drop, weights_hidden2) + biases_hidden2
    hidden2 = tf.nn.relu(score_hidden2)
    hidden2_drop = tf.nn.dropout(hidden2, keep_prob2)
    score_hidden3 = tf.matmul(hidden2_drop, weights_hidden3) + biases_hidden3
    hidden3 = tf.nn.relu(score_hidden3)
    hidden3_drop = tf.nn.dropout(hidden3, keep_prob3)
    
    score = tf.matmul(hidden3_drop, weights_output) + biases_output
    
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(score, tf_train_labels))\
        + (beta1 * tf.nn.l2_loss(weights_hidden1)
           + beta2 * tf.nn.l2_loss(weights_hidden2)
           + beta3 * tf.nn.l2_loss(weights_hidden3)
           + beta4 * tf.nn.l2_loss(weights_output))
    
    # Add learning rate decay
    global_step = tf.Variable(0, trainable=False)
    learning_rate = tf.train.exponential_decay(
        start_learning_rate, global_step, decay_steps, decay_rate, staircase=True)
  
    # Optimizer.
    optimizer = (tf.train.GradientDescentOptimizer(learning_rate)
                   .minimize(loss, global_step=global_step))
  
    # Predictions for the training, validation, and test data.    
    train_prediction = tf.nn.softmax(score)
    
    score_hidden1_valid = tf.matmul(tf_valid_dataset, weights_hidden1) + biases_hidden1
    hidden1_valid = tf.nn.relu(score_hidden1_valid)
    score_hidden2_valid = tf.matmul(hidden1_valid, weights_hidden2) + biases_hidden2
    hidden2_valid = tf.nn.relu(score_hidden2_valid)
    score_hidden3_valid = tf.matmul(hidden2_valid, weights_hidden3) + biases_hidden3
    hidden3_valid = tf.nn.relu(score_hidden3_valid)
    score_valid = tf.matmul(hidden3_valid, weights_output) + biases_output    
    valid_prediction = tf.nn.softmax(score_valid)
    
    score_hidden1_test = tf.matmul(tf_test_dataset, weights_hidden1) + biases_hidden1
    hidden1_test = tf.nn.relu(score_hidden1_test)
    score_hidden2_test = tf.matmul(hidden1_test, weights_hidden2) + biases_hidden2
    hidden2_test = tf.nn.relu(score_hidden2_test)
    score_hidden3_test = tf.matmul(hidden2_test, weights_hidden3) + biases_hidden3
    hidden3_test = tf.nn.relu(score_hidden3_test)
    score_test = tf.matmul(hidden3_test, weights_output) + biases_output    
    test_prediction = tf.nn.softmax(score_test)
with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {tf_train_dataset: batch_data, tf_train_labels: batch_labels}
        _, l, predictions = session.run(
            [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
                  valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))<jupyter_output>Initialized
Minibatch loss at step 0: 2.588271
Minibatch accuracy: 5.5%
Validation accuracy: 32.1%
Minibatch loss at step 500: 0.539551
Minibatch accuracy: 86.7%
Validation accuracy: 85.3%
Minibatch loss at step 1000: 0.669719
Minibatch accuracy: 82.8%
Validation accuracy: 86.4%
Minibatch loss at step 1500: 0.407966
Minibatch accuracy: 93.0%
Validation accuracy: 87.5%
Minibatch loss at step 2000: 0.422861
Minibatch accuracy: 91.4%
Validation accuracy: 87.7%
Minibatch loss at step 2500: 0.458190
Minibatch accuracy: 88.3%
Validation accuracy: 88.1%
Minibatch loss at step 3000: 0.505663
Minibatch accuracy: 86.7%
Validation accuracy: 88.5%
Minibatch loss at step 3500: 0.506377
Minibatch accuracy: 85.2%
Validation accuracy: 89.2%
Minibatch loss at step 4000: 0.418958
Minibatch accuracy: 92.2%
Validation accuracy: 89.2%
Minibatch loss at step 4500: 0.436489
Minibatch accuracy: 89.1%
Validation accuracy: 89.2%
Minibatch loss at step 5000: 0.471058
Minibatch accuracy: 90.6%
Validation accuracy[...]
{'fraction_non_alphanumeric': 0.09907421460009105, 'fraction_numerical': 0.04106844741235392, 'mean_word_length': 3.701869559012416, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '94322', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 11457, 'n_tokens_neox': 10451, 'n_words': 2736, 'path': 'notebook/3_regularization_sol.ipynb', 'repo_name': 'bowen0701/udacity_google_deep_learning', 'size': '32945'}