<jupyter_start><jupyter_text>Exploring DNN learning with TensorFlowIn this assignment we'll dive a little deeper with a series of hands on exercises to better understand DNN learning with Tensorflow. Remember that if you are taking the class for a certificate we will be asking you questions about the assignment in the test!We start by setting up the problem for you.<jupyter_code>import tensorflow as tf

# Load in fashion MNIST
mnist = tf.keras.datasets.fashion_mnist
(training_images, training_labels), (test_images, test_labels) = mnist.load_data()

# Define the base model
model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)), 
                                    tf.keras.layers.Dense(512, activation=tf.nn.relu), 
                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])<jupyter_output><empty_output><jupyter_text>Neural Networks learn the best when the data is scaled / normalized to fall in a constant range. One practitioners often use is the range [0,1]. How might you do this to the training and test images used here?*A hint: these images are saved in the standard [RGB](https://www.rapidtables.com/web/color/RGB_Color.html) format*<jupyter_code>training_images  = training_images / 255.0 #YOUR CODE HERE#
test_images = test_images / 255.0 #YOUR CODE HERE#<jupyter_output><empty_output><jupyter_text>Using these improved images lets compile our model using an adaptive optimizer to learn faster and a categorical loss function to differentiate between the the various classes we are trying to classify. Since this is a very simple dataset we will only train for 5 epochs.<jupyter_code># compile the model
model.compile(optimizer = tf.keras.optimizers.Adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics=['accuracy'])

# fit the model to the training data
model.fit(training_images, training_labels, epochs=5)

# test the model on the test data
model.evaluate(test_images, test_labels)<jupyter_output>Epoch 1/5
1875/1875 [==============================] - 8s 4ms/step - loss: 0.5855 - accuracy: 0.7934
Epoch 2/5
1875/1875 [==============================] - 9s 5ms/step - loss: 0.3587 - accuracy: 0.8707
Epoch 3/5
1875/1875 [==============================] - 8s 4ms/step - loss: 0.3276 - accuracy: 0.8784
Epoch 4/5
1875/1875 [==============================] - 8s 5ms/step - loss: 0.3012 - accuracy: 0.8893
Epoch 5/5
1875/1875 [==============================] - 8s 4ms/step - loss: 0.2820 - accuracy: 0.8950
313/313 [==============================] - 1s 2ms/step - loss: 0.3496 - accuracy: 0.8739<jupyter_text>Once it's done training -- you should see an accuracy value at the end of the final epoch. It might look something like 0.8658. This tells you that your neural network is about 89% accurate in classifying the training data. I.E., it figured out a pattern match between the image and the labels that worked 89% of the time. But how would it work with unseen data? That's why we have the test images. We can call ```model.evaluate```, and pass in the two sets, and it will report back the loss for each. This should reach about .8747 or thereabouts, showing about 87% accuracy. Not Bad! But what did it actually learn? If we inference on the model using ```model.predict``` we get out the following list of values. **What does it represent?***A hint: trying running ```print(test_labels[0])```*<jupyter_code>classifications = model.predict(test_images)
print(classifications[0])<jupyter_output>[6.0692850e-07 3.2736416e-10 8.9253854e-10 8.6722611e-09 3.7620809e-09
 5.6268070e-03 7.8516734e-08 7.0337132e-03 2.7305427e-07 9.8733854e-01]<jupyter_text>Let's now look at the layers in your model. What happens if you double the number of neurons in the dense layer. What different results do you get for loss, training time etc? Why do you think that's the case?<jupyter_code>NUMBER_OF_NEURONS =  1024 #YOUR_CODE_HERE#

# define the new model
model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),
                                    tf.keras.layers.Dense(NUMBER_OF_NEURONS, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])

# compile fit and evaluate the model again
model.compile(optimizer = tf.keras.optimizers.Adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.fit(training_images, training_labels, epochs=5)
model.evaluate(test_images, test_labels)<jupyter_output>Epoch 1/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.5837 - accuracy: 0.7896
Epoch 2/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.3685 - accuracy: 0.8670
Epoch 3/5
1875/1875 [==============================] - 17s 9ms/step - loss: 0.3243 - accuracy: 0.8807
Epoch 4/5
1875/1875 [==============================] - 18s 10ms/step - loss: 0.3010 - accuracy: 0.8875
Epoch 5/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.2724 - accuracy: 0.8979
313/313 [==============================] - 1s 3ms/step - loss: 0.3393 - accuracy: 0.8790<jupyter_text>Consider the effects of additional layers in the network instead of simply more neurons to the same layer. First update the model to add an additional dense layer into the model between the two existing Dense layers.<jupyter_code>YOUR_NEW_LAYER = tf.keras.layers.Dense(512,activation=tf.nn.relu) #YOUR_CODE_HERE#

model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),
                                    tf.keras.layers.Dense(512, activation=tf.nn.relu),
                                    YOUR_NEW_LAYER,
                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])<jupyter_output><empty_output><jupyter_text>Lets then compile, fit, and evaluate our model. What happens to the error? How does this compare to the original model and the model with double the number of neurons?<jupyter_code># compile fit and evaluate the model again
model.compile(optimizer = tf.keras.optimizers.Adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.fit(training_images, training_labels, epochs=5)
model.evaluate(test_images, test_labels)<jupyter_output>Epoch 1/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.5746 - accuracy: 0.7962
Epoch 2/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.3585 - accuracy: 0.8669
Epoch 3/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.3168 - accuracy: 0.8816
Epoch 4/5
1875/1875 [==============================] - 16s 8ms/step - loss: 0.2944 - accuracy: 0.8893
Epoch 5/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.2748 - accuracy: 0.8959
313/313 [==============================] - 1s 3ms/step - loss: 0.3464 - accuracy: 0.8762<jupyter_text>Before you trained, you normalized the data. What would be the impact of removing that? To see it for yourself fill in the following lines of code to get a non-normalized set of data and then re-fit and evaluate the model using this data.<jupyter_code># get new non-normalized mnist data
training_images_non = training_images * 255 #YOUR_CODE_HERE#
test_images_non = test_images * 255 #YOUR_CODE_HERE#

# re-compile, re-fit and re-evaluate
model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),
                                    tf.keras.layers.Dense(512, activation=tf.nn.relu),
                                    YOUR_NEW_LAYER,
                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])
model.compile(optimizer = tf.keras.optimizers.Adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.fit(training_images_non, training_labels, epochs=5)
model.evaluate(test_images_non, test_labels)
classifications = model.predict(test_images_non)<jupyter_output>Epoch 1/5
1875/1875 [==============================] - 16s 8ms/step - loss: 5.2578 - accuracy: 0.6164
Epoch 2/5
1875/1875 [==============================] - 16s 9ms/step - loss: 0.6143 - accuracy: 0.7758
Epoch 3/5
1875/1875 [==============================] - 16s 8ms/step - loss: 0.5053 - accuracy: 0.8202
Epoch 4/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.4921 - accuracy: 0.8260
Epoch 5/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.4468 - accuracy: 0.8447
313/313 [==============================] - 1s 3ms/step - loss: 0.4586 - accuracy: 0.8393<jupyter_text>Sometimes if you set the training for too many epochs you may find that training stops improving and you wish you could quit early. Good news, you can! TensorFlow has a function called ```Callbacks``` which can check the results from each epoch. Modify this callback function to make sure it exits training early but not before reaching at least the second epoch!*A hint: logs.get(METRIC_NAME) will return the value of METRIC_NAME at the current step*<jupyter_code># define and instantiate your custom Callback
class myCallback(tf.keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs={}):
    if(logs.get("accuracy")>0.86):
      self.model.stop_training = True
callbacks = myCallback()

# re-compile, re-fit and re-evaluate
model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),
                            tf.keras.layers.Dense(512, activation=tf.nn.relu),
                            YOUR_NEW_LAYER,
                            tf.keras.layers.Dense(10, activation=tf.nn.softmax)])
model.compile(optimizer = tf.keras.optimizers.Adam(),
      loss = 'sparse_categorical_crossentropy',
      metrics=['accuracy'])
model.fit(training_images, training_labels, epochs=5, callbacks=[callbacks])<jupyter_output>Epoch 1/5
1875/1875 [==============================] - 17s 8ms/step - loss: 0.5810 - accuracy: 0.7937
Epoch 2/5
1875/1875 [==============================] - 16s 9ms/step - loss: 0.3657 - accuracy: 0.8636
{'fraction_non_alphanumeric': 0.17358339984038307, 'fraction_numerical': 0.07631683958499601, 'mean_word_length': 3.845335911068149, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '509802', 'license': 'MIT', 'n_tokens_mistral': 3647, 'n_tokens_neox': 3064, 'n_words': 925, 'path': 'ML/TinyML_Assignment_2_2_12_Exercise.ipynb', 'repo_name': 'bhupendpatil/Practice', 'size': '10024'}