<jupyter_start><jupyter_text>Handwritten Digit Recognition using a Neural NetworkIn this lab, we're going to train a very simple neural network to recognise hardwritten digits. We're going to use the MNIST database of handwritten digits - http://yann.lecun.com/exdb/mnist/We will use the Keras API for TensorFlow 2 in this exmaple, along with NumPy, the Python library for multi-dimensional numerical computing.Let's now import TensorFlow 2 and NumPy into our Python runtime.<jupyter_code>%tensorflow_version 2.x
import tensorflow as tf

import numpy as np<jupyter_output><empty_output><jupyter_text>To ensure these labs run as fast as possible, from the menu above select **Edit > Notebook settings** or **Runtime > Change runtime type** and select GPU as the Hardware accelerator option.Let's test that we are running using the GPU.<jupyter_code>tf.test.gpu_device_name()<jupyter_output><empty_output><jupyter_text>**If** this outputs '', then we are running on CPU only. If it outputs something like '/device:GPU:0' then we are running on GPU. If you see something like ...    Failed to assign a backend    No backend with GPU available. WOuld you like to use a runtime with no accelerator?This suggests that many other users have all the GPU resources on colab occupied at the moment, so perhaps try later or try using with the TPU instead.  Loading our Sample DataWe're going to next download and load the MNIST database of handwritten digits, using TF2.0/Keras helper functions.  This database is already conveniently split into labelled training and test sets for us - there are 60,000 training examples, and 10,000 test examples.We're going to further split the 10,000 test examples into 5,000 test set examples and 5,000 validation set examples.<jupyter_code>mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()

print("%d training examples, %d test examples by default" % (len(x_train), len(x_test)))
# We'll split our test data into test plus validation sets

x_val = x_test[-5000:]
y_val = y_test[-5000:]

x_test = x_test[:-5000]
y_test = y_test[:-5000]

print("After splitting into test and validation sets, we have %d test examples and %d validation examples" % (len(x_test), len(x_val)))<jupyter_output><empty_output><jupyter_text>Let's look at one of these examples - let's say `x_train[0]`...<jupyter_code>import matplotlib.pyplot as plt
# this next line is just Jupyter Notebook magic to plot inline in the webpage...
%matplotlib inline 

exemplar_index = 0

plt.imshow(x_train[exemplar_index])
plt.title("Training Example 0 has label " + str(y_train[exemplar_index]))<jupyter_output><empty_output><jupyter_text>You can change the value of variable `exemplar_index` above and re-run the cell to see the data in the training set and its associated ground truth label.With MNIST, the examples are stored in a simple file format of pixels, where 0 means background and 255 means foreground. We're next going to scale our input to be in the range of 0.0 to 1.0 by dividing by 255.This achieves two things - it takes our precision from integer to 32-bit float, and it also scales our input data so that our default learning rate (and other hyperparameters) will work well.  This step is not necessary, but it does help speed things up.Remember, if you do pre-scaling like this on your training data, you must perform the same steps to any data prior to attempting prediction on it.<jupyter_code>x_train, x_test, x_val = x_train/255.0, x_test / 255.0, x_val / 255.0<jupyter_output><empty_output><jupyter_text>Creating and Configuring our ModelWe're create a few variables that can be used to store important model parameters and hyper parameters for modification layer on.<jupyter_code>num_epochs = 10
dropout_rate = 0.2<jupyter_output><empty_output><jupyter_text>Now, we will create our fully connected neural network model. This isn't a convolutional model, but a fully connected model, and as you'll see it is quite simple - with only a few layers, and approximately 100,000 trainable parameters.<jupyter_code>model = tf.keras.models.Sequential([
  # our input layer will take 28x28 images, and "flatten" them to 784x1
  tf.keras.layers.Flatten(input_shape=(28, 28), name="flatten"), 

  # next, we'll have a densely connected layer using ReLU as its activation function
  tf.keras.layers.Dense(128, activation='relu', name="fc_with_ReLU"), 

  # we'll add some dropout, which is a form of regularisation - 
  # you can experiment with removing this and seeing what difference it makes
  tf.keras.layers.Dropout(dropout_rate, name="dropout"), 

  # finally, a densely connected layer with softmax as a classifier
  tf.keras.layers.Dense(10, activation='softmax', name="fc_softmax_classifier")
])<jupyter_output><empty_output><jupyter_text>You can find out more about the types of layers supported in the Keras API at https://keras.io/layers/core/Let's have a quick look at the model we just created...<jupyter_code>print(model.summary(), '\n')

tf.keras.utils.plot_model(model)<jupyter_output><empty_output><jupyter_text>Training our ModelOkay, now time to compile the model, and to training it.Compiling, in Keras, configures the model for training - it defines the loss function, the optimizer and metrics. It doesn't affect or modify the weights, and you can compile a model as often as you want without affecting pretrained weights.<jupyter_code>model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Just before we training our model ("fit" it to our data), it will be initialised with random weights. Let's see how it performs against our training data *before* we train it...<jupyter_code>model.evaluate(x_train,  y_train, verbose=2)<jupyter_output><empty_output><jupyter_text>Not terribly impressive - but that is just from random weight assigments. Let's now optimize our weights to improve our prediction score against our training data and see what happens.<jupyter_code>history = model.fit(x_train, y_train, validation_data = (x_val, y_val), epochs=num_epochs, shuffle=True)<jupyter_output><empty_output><jupyter_text>In the dictionary variable `history`, we have a record of the training and test set<jupyter_code>print(history.history.keys())<jupyter_output><empty_output><jupyter_text>Evaluating the Performance of our Trained Model against our Validation SetOkay, let's now see how well our trained model works on our validation set.<jupyter_code>model.evaluate(x_val,  y_val, verbose=2)<jupyter_output><empty_output><jupyter_text>At this point, consider going back and changing the `num_epochs` from 5 to, say, 25 and repeating the exercise.  Does our model look like it is overfitting at any point?Or changing the `dropout_rate` from 0.2 to, say, 0.1 or 0.3.  Evaluating the Performance against our Test SetWe reserved our test set to better understand the impact our training and hyper parameter experiments had in generalising to unseen data. We can now evaluate our performance against our test set.We can evaluate our model against our test set and get a figure for overall accuracy.<jupyter_code>model.evaluate(x_test, y_test, verbose=2)<jupyter_output><empty_output><jupyter_text>We can also plot our history from training, and view our training curves - that is, the record of our training set accuracy/loss vs our test set accuracy/loss per epoch.<jupyter_code>plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'])
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model Loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'])
plt.show()<jupyter_output><empty_output><jupyter_text>And we can use `model.predict()` to make new predictions for any unseen data.<jupyter_code>exemplar_index = 0

plt.imshow(x_val[exemplar_index])

predictions = model.predict(x_val)

print("For item %d, our prediction scores across all classes (\"0\" through \"9\") are %s\n" 
      % (exemplar_index, np.array_str(predictions[exemplar_index])))

#
# We're going to select the column corresponding to the highest prediction
# np.amax picks the highest value in an array, and np.where identifies the column
guess = np.where(predictions[exemplar_index] == np.amax(predictions[exemplar_index]))
# the shape of guess from the previous operation is a 1x1 array, so we need to 
# convert this to a scalar
guess = np.squeeze(guess, axis=(0,1)).item()

print("Our best prediction is %d, and the ground truth is %d." 
      % (guess, y_val[exemplar_index]))<jupyter_output><empty_output><jupyter_text>Experiments 1. If you haven't already, try changing `num_epochs` above from 5 to, say, 25 and re-training and re-evaluating.  Can you see the different in accuracy for training for longer? Does our model look like it is overfitting at any point?2. Try changing the `dropout_rate` from 0.2 to, say, 0.1 or 0.3. What is the effect?3. Try commenting out the dropout layer entirely from our model definition above, and re-running. Can you see the impact of the dropout layer in our performance on the test data?      tf.keras.layers.Dropout(dropout_rate, name="dropout"),  we'll add some dropout4. Try scanning some of your own handwriting for a digit, and storing it as 28x28 pixel image. Then load the image, and perform a prediction on it. The following code will allow you to upload it to Colab and run a prediction on it... (**Note:** somethings `file.upload()` method can be flakey - if you get an error along the lines of `MessageError: TypeError: Cannot read property '_uploadFiles' of undefined`, run the following cell again...)<jupyter_code># upload an image file
from google.colab import files

uploaded_image = None

uploaded_image = files.upload()

import io
import cv2

if (uploaded_image != {}):
  # convert the uploaded image data into an OpenCV grayscale inverted image,
  # 28x28, as used by the MNIST data
  image_stream = io.BytesIO(uploaded_image[list(uploaded_image.keys())[0]])                        
  image = cv2.imdecode(np.frombuffer(image_stream.read(), np.uint8),1)
  image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2GRAY)
  image = cv2.bitwise_not(image) # invert image
  image = cv2.resize(image, (28,28))

  plt.imshow(image)

  # now, we're rescale our image from 0-255 to 0.0-1.0, and reshape it as an
  # array for input into model.predict
  model_input = image/255.0
  model_input = np.reshape(model_input, (1, 28, 28))
  predictions = model.predict(model_input)

  # finally, we'll print out all our prediction scores and
  # our best guess for the digit
  print("Our prediction scores across all classes (\"0\" through \"9\") are %s\n" 
      % (np.array_str(predictions[0])))
  guess = np.where(predictions[0] == np.amax(predictions[0]))
  guess = np.squeeze(guess, axis=(0,1)).item()
  print("Our best prediction is %d" % (guess))
else:
  print("An image was not successfully uploaded - try again!")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08830141174354825, 'fraction_numerical': 0.01600575487815844, 'mean_word_length': 5.154952960708356, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 71, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '597581', 'license': 'MIT', 'n_tokens_mistral': 3501, 'n_tokens_neox': 3272, 'n_words': 1322, 'path': 'Lab_4_1_MNIST_with_ANN.ipynb', 'repo_name': 'mabubaker1947/CE6003', 'size': '11121'}