<jupyter_start><jupyter_text>Deep Learning    Lab Session 2 - 1.5 Hours  Convolutional Neural Network (CNN) for Handwritten Digits Recognition  Group name: Deeplearn 38 - <NAME> & <NAME>  The aim of this session is to practice with Convolutional Neural Networks. Each group should fill and run appropriate notebook cells. Generate your final report (export as HTML) and upload it on the submission website http://bigfoot-m1.eurecom.fr/teachingsub/login (using your deeplearnXX/password). Do not forget to run all your cells before generating your final report and do not forget to include the names of all participants in the group. The lab session should be completed and submitted by May 30th 2018 (23:59:59 CET).  Introduction In the previous Lab Session, you built a Multilayer Perceptron for recognizing hand-written digits from the MNIST data-set. The best achieved accuracy on testing data was about 97%. Can you do better than these results using a deep CNN ?In this Lab Session, you will build, train and optimize in TensorFlow one of the early Convolutional Neural Networks,  **LeNet-5**, to go to more than 99% of accuracy.   Load MNIST Data in TensorFlowRun the cell below to load the MNIST data that comes with TensorFlow. You will use this data in **Section 1** and **Section 2**.<jupyter_code>import tensorflow as tf
import numpy as np
from numpy import array
from tensorflow.examples.tutorials.mnist import input_data
from tensorflow.contrib.layers import flatten
from __future__ import print_function
import time
import warnings
warnings.filterwarnings('ignore')

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
X_train, y_train           = mnist.train.images, mnist.train.labels
X_validation, y_validation = mnist.validation.images, mnist.validation.labels
X_test, y_test             = mnist.test.images, mnist.test.labels
print("Image Shape: {}".format(X_train[0].shape))
print("Training Set:   {} samples".format(len(X_train)))
print("Validation Set: {} samples".format(len(X_validation)))
print("Test Set:       {} samples".format(len(X_test)))

epsilon = 1e-10 # this is a parameter you will use later<jupyter_output>Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
Image Shape: (784,)
Training Set:   55000 samples
Validation Set: 5000 samples
Test Set:       10000 samples<jupyter_text>Section 1 : My First Model in TensorFlow Before starting with CNN, let's train and test in TensorFlow the example**y=softmax(Wx+b)** seen in the first lab. This model reaches an accuracy of about 92 %.You will also learn how to launch the TensorBoard https://www.tensorflow.org/get_started/summaries_and_tensorboard to visualize the computation graph, statistics and learning curves.   Part 1  : Read carefully the code in the cell below. Run it to perform training.<jupyter_code>#STEP 1

# Parameters
learning_rate = 0.01
training_epochs = 40
batch_size = 128
display_step = 1
logs_path = 'log_files/'  # useful for tensorboard

# tf Graph Input:  mnist data image of shape 28*28=784
x = tf.placeholder(tf.float32, [None, 784], name='InputData')
# 0-9 digits recognition,  10 classes
y = tf.placeholder(tf.float32, [None, 10], name='LabelData')

# Set model weights
W = tf.Variable(tf.zeros([784, 10]), name='Weights')
b = tf.Variable(tf.zeros([10]), name='Bias')

# Construct model and encapsulating all ops into scopes, making Tensorboard's Graph visualization more convenient
with tf.name_scope('Model'):
    # Model
    pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax
with tf.name_scope('Loss'):
    # Minimize error using cross entropy
    # We use tf.clip_by_value to avoid having too low numbers in the log function
    cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(tf.clip_by_value(pred, epsilon, 1.0)), reduction_indices=1))
with tf.name_scope('SGD'):
    # Gradient Descent
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
with tf.name_scope('Accuracy'):
    # Accuracy
    acc = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
    acc = tf.reduce_mean(tf.cast(acc, tf.float32))

# Initializing the variables
init = tf.global_variables_initializer()
# Create a summary to monitor cost tensor
tf.summary.scalar("Loss", cost)
# Create a summary to monitor accuracy tensor
tf.summary.scalar("Accuracy", acc)
# Merge all summaries into a single op
merged_summary_op = tf.summary.merge_all()

#STEP 2 

# Launch the graph for training
with tf.Session() as sess:
    sess.run(init)

    # op to write logs to Tensorboard
    summary_writer = tf.summary.FileWriter(logs_path, graph=tf.get_default_graph())
    # Training cycle
    for epoch in range(training_epochs):
        avg_cost = 0.
        total_batch = int(mnist.train.num_examples/batch_size)
        # Loop over all batches
        for i in range(total_batch):
            batch_xs, batch_ys = mnist.train.next_batch(batch_size, shuffle=(i==0))
            # Run optimization op (backprop), cost op (to get loss value)
            # and summary nodes
            _, c, summary = sess.run([optimizer, cost, merged_summary_op],
                                     feed_dict={x: batch_xs, y: batch_ys})
            # Write logs at every iteration
            summary_writer.add_summary(summary, epoch * total_batch + i)
            # Compute average loss
            avg_cost += c / total_batch
        # Display logs per epoch step
        if (epoch+1) % display_step == 0:
            print("Epoch: ", '%02d' % (epoch+1), "  =====> Loss=", "{:.9f}".format(avg_cost))

    print("Optimization Finished!")
    summary_writer.flush()

    # Test model
    # Calculate accuracy
    print("Accuracy:", acc.eval({x: mnist.test.images, y: mnist.test.labels}))<jupyter_output>Epoch:  01   =====> Loss= 1.288166094
Epoch:  02   =====> Loss= 0.732756539
Epoch:  03   =====> Loss= 0.600399079
Epoch:  04   =====> Loss= 0.536668742
Epoch:  05   =====> Loss= 0.497883010
Epoch:  06   =====> Loss= 0.471130213
Epoch:  07   =====> Loss= 0.451441533
Epoch:  08   =====> Loss= 0.435906629
Epoch:  09   =====> Loss= 0.423306856
Epoch:  10   =====> Loss= 0.413005027
Epoch:  11   =====> Loss= 0.404336038
Epoch:  12   =====> Loss= 0.396785073
Epoch:  13   =====> Loss= 0.390349418
Epoch:  14   =====> Loss= 0.384353908
Epoch:  15   =====> Loss= 0.379181639
Epoch:  16   =====> Loss= 0.374695726
Epoch:  17   =====> Loss= 0.370589093
Epoch:  18   =====> Loss= 0.366279961
Epoch:  19   =====> Loss= 0.362964873
Epoch:  20   =====> Loss= 0.359622410
Epoch:  21   =====> Loss= 0.356603251
Epoch:  22   =====> Loss= 0.353947618
Epoch:  23   =====> Loss= 0.351042153
Epoch:  24   =====> Loss= 0.348606701
Epoch:  25   =====> Loss= 0.346464758
Epoch:  26   =====> Loss= 0.344220706
Epoch:  27  [...]<jupyter_text>COMMENT:We just trained and tested the multilayer perceptron model using softmax.The accuracy we get is 0.9157.This is already a good accuray (above 90% !) but we're aiming a 99% accuracy, and this will be our challenge in section 2.  Part 2  : Using Tensorboard, we can  now visualize the created graph, giving you an overview of your architecture and how all of the major components  are connected. You can also see and analyse the learning curves. To launch tensorBoard: - Open a Terminal and run the command line **"tensorboard --logdir=lab_2/log_files/"**- Click on "Tensorboard web interface" in Zoe  Enjoy It !!   Section 2 : The 99% MNIST Challenge !  Part 1  : LeNet5 implementation You are now more familar with **TensorFlow** and **TensorBoard**. In this section, you are to build, train and test the baseline [LeNet-5](http://yann.lecun.com/exdb/lenet/)  model for the MNIST digits recognition problem.  Then, you will make some optimizations to get more than 99% of accuracy.For more informations, have a look at this list of results: http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html ![alt text](lenet.png "Figure 1: Lenet-5") Figure 1: Lenet-5 The LeNet architecture takes a 28x28xC image as input, where C is the number of color channels. Since MNIST images are grayscale, C is 1 in this case.--------------------------**Layer 1 - Convolution (5x5):** The output shape should be 28x28x6. **Activation:** ReLU. **MaxPooling:** The output shape should be 14x14x6.**Layer 2 - Convolution (5x5):** The output shape should be 10x10x16. **Activation:** ReLU. **MaxPooling:** The output shape should be 5x5x16.**Flatten:** Flatten the output shape of the final pooling layer such that it's 1D instead of 3D.  You may need to use tf.reshape.**Layer 3 - Fully Connected:** This should have 120 outputs. **Activation:** ReLU.**Layer 4 - Fully Connected:** This should have 84 outputs. **Activation:** ReLU.**Layer 5 - Fully Connected:** This should have 10 outputs. **Activation:** softmax.  Question 2.1.1   Implement the Neural Network architecture described above.For that, your will use classes and functions from  https://www.tensorflow.org/api_docs/python/tf/nn. We give you some helper functions for weigths and bias initilization. Also you can refer to section 1.<jupyter_code># Functions for weigths and bias initilization 
def weight_variable(shape):
  initial = tf.truncated_normal(shape, stddev=0.1)
  return tf.Variable(initial)

def bias_variable(shape):
  initial = tf.constant(0., shape=shape)
  return tf.Variable(initial)
# We define LeNet-5 convolutional and max pool layers
def conv2d(x, W,pad='SAME'):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding=pad)
# 2x2 max pooling layer
def max_pool_size2(x,pad='VALID'):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding=pad)
def LeNet5_Model(data, keep_prob=1):
    input_data = tf.reshape(data,[-1,28,28,1])    
    
    # Layer 1: Convolution / Input shape = 28x28x1 / Output shape = 28x28x6
    w1 = weight_variable([5,5,1,6])
    b1 = bias_variable([6])
    conv1 = tf.nn.relu(conv2d(input_data , w1) + b1) #activation
    
    # Max Pooling / Input shape = 28x28x6 / Output shape = 14x14x6
    pool1 = max_pool_size2(conv1)
    
    # Layer 2: Convolutional / Input shape = 14*14*6 / Output shape = 10x10x16
    w2 = weight_variable([5,5,6,16])
    b2 = bias_variable([16])
    conv2 = tf.nn.relu(conv2d( pool1  ,  w2  ,  pad='VALID') + b2) # activation
    
    # Max Pooling / Input shape = 10x10x16 / Output shape = 5x5x16
    pool2 = max_pool_size2(conv2)
    
    # Flatten the output shape / Input shape = 5x5x16 / Output shape = 400
    pool2_flat = tf.contrib.layers.flatten(pool2)
    
    # Layer 3: Fully Connected / Input shape = 400 / Output shape = 120
    w3 = weight_variable([400, 120])
    b3 = bias_variable([120])
    fc1 = tf.nn.relu(tf.matmul(pool2_flat, w3) + b3) # activation
    
    # Layer 4: Fully Connected / Input shape = 120 / Output shape = 84
    w4 = weight_variable([120, 84])
    b4 = bias_variable([84])
    fc2 = tf.nn.relu(tf.matmul(fc1, w4) + b4) # activation
    
    # Layer 5: Fully Connected / Input shape = 84 / Output shape = 10
    w5 = weight_variable([84, 10])
    b5 = bias_variable([10])
    
    pred = tf.nn.softmax(tf.matmul(fc2, w5) + b5) # softmax activation
    
    return pred<jupyter_output><empty_output><jupyter_text>Question 2.1.2.   Calculate the number of parameters of this model<jupyter_code>number_parameters_LeNet5 = (5*5*1 + 1)*6 + (5*5*6 + 1)*16 + (400 + 1)*120 + (120 + 1)*84 + (84 + 1)*10
print('Number of parameters in LeNet5: {}'.format(number_parameters_LeNet5))<jupyter_output>Number of parameters in LeNet5: 61706<jupyter_text>Explanations:The input layer only reads the input images, so there is no parameter to learn in this layer.For the first convolutionnal layer we have 1 filter of size 5 x 5. The output size is 6, so we learn 6 different 5x5x1 filters. There is also a bias term for each output, so we have a total number of parameters of:    Weights and biases for convolutional layer 1 = (5 x 5 x 1 +1) x 6The reasoning is the same for the second convolutional layer:    Weights and biases for convolutional layer 2 = (5 x 5 x 6 +1) x 16There is no parameter to learn in a pooling layer.In a fully-connected laye, all input units have a separate weight to each output unit. For n inputs and m outputs, the number of weights is n x m. Additionally, there is a bias for each output node, so it comes to (n+1) x m parameters.    Weights and biases for fully connected layer 1 = (400 + 1) x 120        Weights and biases for fully connected layer 2 = (120 + 1) x 84The output layer is just a normal fully connected layer. The same formula applies.    Weights and biases for output layer = (84 + 1) x 10TOTAL number of parameters: 61706  Question 2.1.3.   Define your model, its accuracy and the loss function according to the following parameters (you can look at Section 1 to see what is expected):     Learning rate: 0.001     Loss Fucntion: Cross-entropy     Optimizer: tf.train.GradientDescentOptimizer     Number of epochs: 40     Batch size: 128<jupyter_code>def apply_le_net ( modelName, model, learning_rate = 0.001, 
            training_epochs = 40, batch_size = 128, display_step = 1, keep_p = 1,
            optimFunc = tf.train.GradientDescentOptimizer,
            X_train=mnist.train.images, y_train=mnist.train.labels,
            X_val=mnist.validation.images, y_val=mnist.validation.labels,
            X_test= mnist.test.images,y_test=mnist.test.labels
          ):
    
    InputSize , OutputSize, TrainingSetSize, ValidationSetSize, TestSetSize = X_train[0].shape[0], y_train[0].shape[0], len(X_train), len(X_validation), len(X_test)    
    
    logsFolder = 'log_files/' # useful for tensorboard
    saveFolder = 'Models/'    # useful to restore the model
    
    tf.reset_default_graph() # reset graph

    # tf Graph Input:  mnist data image of shape 28*28*1
    x = tf.placeholder(tf.float32, [None,InputSize], name='InputData')
    # 0-9 digits recognition,  10 classes
    y = tf.placeholder(tf.float32, [None,OutputSize], name='LabelData')
    # Dropout
    keep_prob = tf.placeholder(tf.float32, name='DropoutKeepProbability')

    with tf.name_scope('Model'):
        # Model
        model = model(x, keep_prob)
    
    with tf.name_scope('Loss'):
        # Minimize error using cross entropy
        cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(model+1e-9), reduction_indices=1))
        #cost = tf.nn.softmax_cross_entropy_with_logits(model, tf.one_hot(y, 10))
    with tf.name_scope('Optimizer'):
        #Optimization, using cost reduction
        optimizer = optimFunc(learning_rate).minimize(cost)
        
    with tf.name_scope('Accuracy'):
        # Accuracies
        acc = let_net_accuracy(model, y)
          
    init = tf.global_variables_initializer()


    # track accuracies
    # Create a summary to monitor cost tensor
    tf.summary.scalar("BatchLoss", cost)
    # Create a summary to monitor batch accuracy tensor
    tf.summary.scalar("BatchAccuracy", acc)
    # Merge all summaries into a single op
    merged_summary_op = tf.summary.merge_all()
    
    # train the model
    train_le_net(model, cost, optimizer, acc,
             x, y, keep_prob, TrainingSetSize,
             X_train, y_train, X_val, y_val, X_test, y_test,
             init, merged_summary_op,
             modelName, saveFolder,logsFolder,
             learning_rate, training_epochs, batch_size, display_step , keep_p)<jupyter_output><empty_output><jupyter_text>Question 2.1.4.   Implement the evaluation function for accuracy computation<jupyter_code>def let_net_accuracy(model, y):
    accuracy = tf.reduce_mean(tf.cast( tf.equal( tf.argmax(model, 1), tf.argmax(y, 1)), tf.float32))
    return accuracy<jupyter_output><empty_output><jupyter_text>Question 2.1.5.   Implement training pipeline and run the training data through it to train the model.- Before each epoch, shuffle the training set. - Print the loss per mini batch and the training/validation accuracy per epoch. (Display results every 100 epochs)- Save the model after training- Print after training the final testing accuracy<jupyter_code>def train_le_net(model, cost, optimizer, acc,
             x, y, keep_prob, TrainingSetSize,
             X_train, y_train, X_val, y_val, X_test, y_test,
             init, merged_summary_op,
             modelName, saveFolder, logsFolder,
             learning_rate, training_epochs, batch_size, display_step, keep_p,
             dataFileName=None):
    
    # Initial model print
    print("*Model ", modelName," {learning rate: %.4f; nb of iterations: %d; batch size: %d}"%\
          (learning_rate, training_epochs, batch_size))
    
    # Start a tensorflow session
    with tf.Session() as sess:
        print ("   Start Training!")
        sess.run(init)
        
        saver = tf.train.Saver()
        """ # Load model if the parameter loadModel is not empty
        if(loadModel):
            print(loadModel)
            saver.restore(sess=sess,save_path='Models/'+str(loadModel))"""
       
        
        # op to write logs to Tensorboard
        summary_writer = tf.summary.FileWriter(logsFolder,
                                               graph=tf.get_default_graph())
        
        # Training cycle
        t0 = time.time()
        for epoch in range(training_epochs):
            avg_cost = 0.
            total_batch = int(mnist.train.num_examples/batch_size)
            # Loop over all batches
            for i in range(total_batch):
                
                batch_xs, batch_ys = mnist.train.next_batch(batch_size)
                """batch_xs, batch_ys = input_pipeline(filenames=dataFileName, 
                                                    batch_size=batch_size, 
                                                    num_epochs=total_batch)"""
                # Run optimization op (backprop), cost op (to get loss value)
                # and summary nodes
                _, c, summary = sess.run([optimizer, cost, merged_summary_op],
                                          feed_dict={x: batch_xs,
                                                     y: batch_ys,
                                                     keep_prob: keep_p})
                # Write logs at every iteration
                summary_writer.add_summary(summary, epoch * total_batch + i)
                # Compute average loss
                avg_cost += c / total_batch

            # Display logs per epoch step
            if (epoch+1) % display_step == 0:
                tr_acc = acc.eval({x: X_train, y: y_train, keep_prob: 1})
                vl_acc = acc.eval({x: X_val, y: y_val, keep_prob: 1})
                print("   Epoch: %02d | Loss=%.9f | Train accuracy=%.3f %% | Validation Accuracy=%.3f %%"% 
                      (epoch+1, avg_cost, tr_acc*100, vl_acc*100));
        
        
        print ("   Training Finished in %.1f seconds."%(time.time()-t0))
        
        # Evaluating model with the accuracies
        print ("      \nFinal accuracies:\n")
        print ("    * Train accuracy: %.3f %%"%(100*acc.eval({x: X_train, y: y_train, keep_prob: 1})))
        print ("    * Validation accuracy: %.3f %%"%(100*acc.eval({x: X_val, y: y_val, keep_prob: 1})))
        print ("    * Test accuracy: %.3f %%"%(100*acc.eval({x: X_test, y: y_test, keep_prob: 1})))
        
        # Saving Model
        saver.save(sess=sess,save_path=saveFolder+modelName)
        print ("   Saving model in file: %s"%(saveFolder+modelName))
apply_le_net (model=LeNet5_Model, modelName ='lenet5-model-SGD-lr=0.001',
           learning_rate = 0.001, 
           training_epochs = 40,
           batch_size = 128, 
           optimFunc = tf.train.GradientDescentOptimizer
      )
apply_le_net (model=LeNet5_Model, modelName ='lenet5-model-SGD-lr=0.01',
           learning_rate = 0.01, 
           training_epochs = 40,
           batch_size = 128,
           optimFunc = tf.train.GradientDescentOptimizer
      )<jupyter_output>*Model  lenet5-model-SGD-lr=0.01  {learning rate: 0.0100; nb of iterations: 40; batch size: 128}
   Start Training!
   Epoch: 01 | Loss=1.422941481 | Train accuracy=86.687 % | Validation Accuracy=87.400 %
   Epoch: 02 | Loss=0.369122338 | Train accuracy=91.024 % | Validation Accuracy=91.340 %
   Epoch: 03 | Loss=0.254931824 | Train accuracy=93.400 % | Validation Accuracy=93.660 %
   Epoch: 04 | Loss=0.201007217 | Train accuracy=94.753 % | Validation Accuracy=95.060 %
   Epoch: 05 | Loss=0.167994749 | Train accuracy=95.029 % | Validation Accuracy=95.440 %
   Epoch: 06 | Loss=0.147176167 | Train accuracy=95.984 % | Validation Accuracy=96.240 %
   Epoch: 07 | Loss=0.131061102 | Train accuracy=96.384 % | Validation Accuracy=96.560 %
   Epoch: 08 | Loss=0.118119375 | Train accuracy=96.695 % | Validation Accuracy=96.800 %
   Epoch: 09 | Loss=0.108751074 | Train accuracy=96.924 % | Validation Accuracy=96.940 %
   Epoch: 10 | Loss=0.102593396 | Train accuracy=97.164 % | Validation Accuracy=97.[...]<jupyter_text>Question 2.1.6  : Use TensorBoard to visualise and save loss and accuracy curves. You will save figures in the folder **"lab_2/MNIST_figures"** and display them in your notebook. Please put your loss and accuracy curves here. ---**Here are the curves of the accuracy and the loss obtained from Tensorboard:** With `learning_rate = 0.001`![alt text](MNIST_figures/final-SGD-001.png "Tensorboard curves - GradientDescent-0.001") With `learning_rate = 0.01`![alt text](MNIST_figures/final-SGD-01.png "Tensorboard curves - GradientDescent-0.01") --- Part 2  : LeNET 5 Optimization  Question 2.2.1 ---- Retrain your network with AdamOptimizer and then fill the table- Which optimizer gives the best accuracy on test data?---<jupyter_code>apply_le_net ( model=LeNet5_Model, modelName='lenet5-model-ADAM-lr=0.001',
           learning_rate = 0.001, 
           training_epochs = 40,
           batch_size = 128, 
           optimFunc = tf.train.AdamOptimizer
      )
apply_le_net (model=LeNet5_Model, modelName='lenet5-model-ADAM-lr=0.01', 
           learning_rate = 0.01, 
           training_epochs = 40,
           batch_size = 128, 
           optimFunc = tf.train.AdamOptimizer
      )<jupyter_output>*Model  lenet5-model-ADAM-lr=0.01  {learning rate: 0.0100; nb of iterations: 40; batch size: 128}
   Start Training!
   Epoch: 01 | Loss=0.163237772 | Train accuracy=98.011 % | Validation Accuracy=97.980 %
   Epoch: 02 | Loss=0.062061921 | Train accuracy=98.102 % | Validation Accuracy=97.760 %
   Epoch: 03 | Loss=0.055803975 | Train accuracy=98.762 % | Validation Accuracy=98.480 %
   Epoch: 04 | Loss=0.050317014 | Train accuracy=99.165 % | Validation Accuracy=98.680 %
   Epoch: 05 | Loss=0.051566172 | Train accuracy=98.704 % | Validation Accuracy=98.380 %
   Epoch: 06 | Loss=0.050115155 | Train accuracy=99.002 % | Validation Accuracy=98.500 %
   Epoch: 07 | Loss=0.039589857 | Train accuracy=99.142 % | Validation Accuracy=98.440 %
   Epoch: 08 | Loss=0.047223618 | Train accuracy=98.962 % | Validation Accuracy=98.420 %
   Epoch: 09 | Loss=0.039301384 | Train accuracy=99.278 % | Validation Accuracy=98.720 %
   Epoch: 10 | Loss=0.049096069 | Train accuracy=99.107 % | Validation Accuracy=98[...]<jupyter_text>---These are the results we have obtained for **`learning rate = 0.001`**| Optimizer                    |  Gradient Descent (learning rate = 0.001) |    AdamOptimizer (learning rate = 0.001) ||------------------------------|-----------------------|-----------------------|| Testing Accuracy             |         95.39 %       |       99.05 %         |       | Training Time (seconds)      |         1365.2        |       1520.1          |  ---These are the results we have obtained for **`learning rate = 0.01`**| Optimizer                    |  Gradient Descent (learning rate = 0.01) |    AdamOptimizer (learning rate = 0.01) ||------------------------------|-----------------------|-----------------------|| Testing Accuracy             |         98.53 %       |       97.01 %         |       | Training Time (seconds)      |         1568.7        |       1521.0          |  ANSWER:In both cases (both learning rates' values), the optimizer which gives the best accuracy on test data is AdamOptimizer. ADAM algorithm both adapts the learning_rate to the parameters, and add momentum (accelerate the gradient in the relevant direction while reducing oscillations).Therefore, for the next question, we will keep this optimizer instead of Gradient Descent. Here are the curves of the accuracy and the loss obtained from Tensorboard: With `learning_rate 0.001`![alt text](MNIST_figures/final-Adam-001.png "Tensorboard curves - ADAM-0.001") With `learning_rate 0.01`![alt text](MNIST_figures/final-Adam-01.png "Tensorboard curves - ADAM-0.01")  Question 2.2.2 Try to add dropout (keep_prob = 0.75) before the first fully connected layer. You will use tf.nn.dropout for that purpose. What accuracy do you achieve on testing data?**Accuracy achieved on testing data:** 99.24 % !<jupyter_code>def LeNet5_Model_Dropout(data, keep_prob=0.75):
    input_data = tf.reshape(data,[-1,28,28,1])
    
    # Layer 1: Convolution / Input shape = 28x28x1 / Output shape = 28x28x6
    w1 = weight_variable([5,5,1,6])
    b1 = bias_variable([6])
    conv1 = tf.nn.relu(conv2d(input_data , w1) + b1) #activation
    
    # Max Pooling / Input shape = 28x28x6 / Output shape = 14x14x6
    pool1 = max_pool_size2(conv1)
    
    # Layer 2: Convolutional / Input shape = 14*14*6 / Output shape = 10x10x16
    w2 = weight_variable([5,5,6,16])
    b2 = bias_variable([16])
    conv2 = tf.nn.relu(conv2d( pool1  ,  w2  ,  pad='VALID') + b2) # activation
    
    # Max Pooling / Input shape = 10x10x16 / Output shape = 5x5x16
    pool2 = max_pool_size2(conv2)
    
    # add dropout here
    drop_pool2 = tf.nn.dropout(pool2, keep_prob)
    
    # Flatten the output shape / Input shape = 5x5x16 / Output shape = 400
    pool2_flat = tf.contrib.layers.flatten(drop_pool2)
    
    # Layer 3: Fully Connected / Input shape = 400 / Output shape = 120
    w3 = weight_variable([400, 120])
    b3 = bias_variable([120])
    fc1 = tf.nn.relu(tf.matmul(pool2_flat, w3) + b3) # activation
    
    # Layer 4: Fully Connected / Input shape = 120 / Output shape = 84
    w4 = weight_variable([120, 84])
    b4 = bias_variable([84])
    fc2 = tf.nn.relu(tf.matmul(fc1, w4) + b4) # activation
    
    # Layer 5: Fully Connected / Input shape = 84 / Output shape = 10
    w5 = weight_variable([84, 10])
    b5 = bias_variable([10])
    
    pred = tf.nn.softmax(tf.matmul(fc2, w5) + b5) # softmax activation
    
    return pred
apply_le_net ('lenet5-model-ADAM-Dropout-lr=0.001', LeNet5_Model_Dropout,
           learning_rate = 0.001, 
           training_epochs = 40,
           batch_size = 128, 
           optimFunc = tf.train.AdamOptimizer
      )
apply_le_net ('lenet5-model-ADAM-Dropout-lr=0.01', LeNet5_Model_Dropout,
           learning_rate = 0.01, 
           training_epochs = 40,
           batch_size = 128, 
           optimFunc = tf.train.AdamOptimizer
      )<jupyter_output>*Model  lenet5-model-ADAM-Dropout-lr=0.01  {learning rate: 0.0100; nb of iterations: 40; batch size: 128}
   Start Training!
   Epoch: 01 | Loss=0.163451382 | Train accuracy=98.120 % | Validation Accuracy=98.000 %
   Epoch: 02 | Loss=0.069643003 | Train accuracy=97.780 % | Validation Accuracy=97.940 %
   Epoch: 03 | Loss=0.058219914 | Train accuracy=98.511 % | Validation Accuracy=98.360 %
   Epoch: 04 | Loss=0.054814242 | Train accuracy=98.847 % | Validation Accuracy=98.640 %
   Epoch: 05 | Loss=0.048273921 | Train accuracy=98.549 % | Validation Accuracy=98.220 %
   Epoch: 06 | Loss=0.045832736 | Train accuracy=98.676 % | Validation Accuracy=98.200 %
   Epoch: 07 | Loss=0.052985071 | Train accuracy=98.687 % | Validation Accuracy=98.280 %
   Epoch: 08 | Loss=0.046835801 | Train accuracy=98.831 % | Validation Accuracy=98.500 %
   Epoch: 09 | Loss=0.042724919 | Train accuracy=99.056 % | Validation Accuracy=98.520 %
   Epoch: 10 | Loss=0.043842827 | Train accuracy=99.109 % | Validation Acc[...]
{'fraction_non_alphanumeric': 0.1150162019727237, 'fraction_numerical': 0.06733611081437169, 'mean_word_length': 3.0234957020057305, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '628451', 'license': 'MIT', 'n_tokens_mistral': 10260, 'n_tokens_neox': 8852, 'n_words': 2685, 'path': 'Lab2-CNN-lenet-and-MNIST-challenge/Lab2_DL-Students-Bonnaud-Nguyen.ipynb', 'repo_name': 'kanguyn/Deep-Learning-spring2018', 'size': '28103'}