<jupyter_start><jupyter_text>Supervised Learning - Classification ExampleBefore reading through this program, it is recommended to make sure you read through the Supervised Learning - Regression article as there will be less basic explanations in this program.   OverviewThis program will make use of the MNIST dataset, a dataset containing handwritten digits, which is known as the Hello World of machine learning. This tutorial will go through creating and training the machine learning model, but later in the series we will use TensorFlow.js to embed this model into an interactive webpage to be able to draw our own digits which can then be classified. Importing Libraries - Matplotlib is a library for creating graphs and other visuals - Numpy is a high-performance mathematics library - TensorFlow is the platform that the AI code will run on, with Keras being the interface to TF<jupyter_code>%load_ext tensorboard

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
from tensorflow import keras
print(tf.__version__)<jupyter_output>2.1.0<jupyter_text>Loading the DatasetInstead of manually downloading the dataset like in the regression example, TensorFlow includes some example dataset which are already formatted that we can use. We make a reference to the MNIST dataset object and load the data which is in the form of 2 tuples of 2 arrays each.We output the shape the train set, it tells us there are 60000 images which are each 28 x 28 pixels in size.<jupyter_code>mnist = keras.datasets.mnist
(trainImages, trainClasses), (testImages, testClasses) = mnist.load_data()
trainImages.shape<jupyter_output><empty_output><jupyter_text>We can also matplotlib to output some example images with their corresponding labels. This block of code creates a figure of (relative) size 10x10, and then iterates through the first 25 training images, creating a subplot for each, and showing the images in black and white with their label.<jupyter_code>plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.imshow(trainImages[i], cmap=plt.cm.binary)
    plt.xlabel(trainClasses[i])
plt.show()<jupyter_output><empty_output><jupyter_text>Normalising the DataWe need to look at how the images are represented, to see if we need to make any changes to how the data is structured. Here we just take the first index in the train Images list, and look at the middle row.<jupyter_code>print(trainImages[0][14])
print("Max Value = " + str(trainImages.max()))
print("Min Value = " + str(trainImages.min()))<jupyter_output>[  0   0   0   0   0   0   0   0   0   0   0   0   0  81 240 253 253 119
  25   0   0   0   0   0   0   0   0   0]
Max Value = 255
Min Value = 0<jupyter_text>Above, we see the values are in the range 0 - 255. The previous article explained that neural networks like to with values that are between 0 - 1 or -1 - 1. Since we know the values won't go below 0, we can normalise the data to be between 0 and 1. This is simply done by dividing every value by 255 (which we know is maximum value):<jupyter_code>trainImages = trainImages / 255
testImages = testImages / 255
trainImages[0][14]<jupyter_output><empty_output><jupyter_text>Building our ModelWe will use a deep neural network using the Keras sequential model. We start with a layer that flattens the 28x28 pixel input into a 784x1 input. We then add a dense layer of size 128 with the reLU activation, with a dropout rate of 0.1.Dropout is used to prevent overfitting which is where the model gets too fixated on the data it is learning from and would not be able to generalise well. This is done by randomly setting some of the weights to 0 during training which forces the model to constantly adapt and change.We then add another dense layer but this time with 32 nodes with reLU.And finally, for the output we have a dense layer with 10 nodes - one for each possible number (0-9)We compile it using the adam optimiser, and to use the Sparce Categorical Cross Entropy loss function (which is a reommended loss function of classification problems). We also include the metrics parameter which will output the accuracy as it trains.We can also run the evaluate function before training to see how well (or badly) it would do without training - it was getting 8-12% accuracy (roughly 1 in 10) which is the probability you would get if you randomly picked an answer each time.<jupyter_code>model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dropout(0.1),
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(10)
])

model.compile(optimizer='adam', loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=['accuracy'])

preTrain = model.evaluate(testImages, testClasses, verbose=2)<jupyter_output>10000/10000 - 0s - loss: 2.3877 - accuracy: 0.0808<jupyter_text>Training our ModelNow the model has been made, we can train it with the train images and train classes, and telling it to run 10 iterations (epochs). By the end of training, this was getting an accuracy of ~0.99 (99% accurate) on the training data!<jupyter_code>model.fit(trainImages, trainClasses, epochs=10)<jupyter_output>Train on 60000 samples
Epoch 1/10
60000/60000 [==============================] - 2s 38us/sample - loss: 0.2768 - accuracy: 0.9179
Epoch 2/10
60000/60000 [==============================] - 2s 34us/sample - loss: 0.1243 - accuracy: 0.9620
Epoch 3/10
60000/60000 [==============================] - 2s 33us/sample - loss: 0.0906 - accuracy: 0.9709
Epoch 4/10
60000/60000 [==============================] - 2s 34us/sample - loss: 0.0731 - accuracy: 0.9771
Epoch 5/10
60000/60000 [==============================] - 2s 34us/sample - loss: 0.0625 - accuracy: 0.9804
Epoch 6/10
60000/60000 [==============================] - 2s 34us/sample - loss: 0.0554 - accuracy: 0.9818
Epoch 7/10
60000/60000 [==============================] - 2s 35us/sample - loss: 0.0474 - accuracy: 0.9845
Epoch 8/10
60000/60000 [==============================] - 2s 34us/sample - loss: 0.0406 - accuracy: 0.9863
Epoch 9/10
60000/60000 [==============================] - 2s 34us/sample - loss: 0.0401 - accuracy: 0.9867
Epoch 10/10
60[...]<jupyter_text>Evaluating our ModelOnce trained, we can evaluate the model using the test images and classes. Here we also use the verbose parameter to just output the results at the end of evaluating - it was getting ~98%. The evaluation accuracy will be usually than the training accuracy due to overfitting like mentioned earlier. Overfitting can be reduced through the user of dropout and by increasing the volume and variation of data - it's very difficult to get exactly the same performance between training and testing.<jupyter_code>testEval = model.evaluate(testImages, testClasses, verbose=2)<jupyter_output>10000/10000 - 0s - loss: 0.0762 - accuracy: 0.9785<jupyter_text>Making PredictionsThe model currently outputs in the form of Logits, we can add softmax layer which transforms the Logit values into typical probabilities that are easier to interpret.Once we create this predictor model, we can then feed it all the testing images to get an array of the predicted values.You could also create your own images and use a library such as Pillow to convert the image into a 28x28 monochrome image that can be fed into the model - this is something we will do with TensorFlow.js in a future tutorial<jupyter_code>predictor = keras.Sequential()
predictor.add(model)
predictor.add(keras.layers.Softmax())
predictions = predictor.predict(testImages)<jupyter_output><empty_output><jupyter_text>With that list of predicted values, we can use matplotlib to visualise the image the model sees, and the output of the model in terms of the probabilities of each valueTo do this, we randomly pick an index of a test image to look at. We create a subplot , plot the image in one plot like we did previously and then create a bar chart<jupyter_code>import random
for count in range(5):
    index = random.randint(0, len(testImages))
    plt.figure(figsize=(6,3))
    
    ## Image
    plt.subplot(1,2,1)
    plt.imshow(testImages[index], cmap=plt.cm.binary)
    plt.xlabel(testClasses[index])
    plt.xticks([])
    plt.yticks([])
    
    ## Prediction Chart
    plt.subplot(1,2,2)
    plt.xticks(range(10))
    plt.yticks([])
    bar = plt.bar(range(10), predictions[index], color='gray')
    bar[np.argmax(predictions[index])].set_color('red')
    bar[testClasses[index]].set_color('green')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10237596086652691, 'fraction_numerical': 0.0528767761472164, 'mean_word_length': 4.623444662737394, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '526547', 'license': 'MIT', 'n_tokens_mistral': 2773, 'n_tokens_neox': 2417, 'n_words': 1086, 'path': '2) Supervised Learning with Classification/SupervisedClassification.ipynb', 'repo_name': 'Cabbit-Cloud/Introduction-to-Machine-Learning', 'size': '8586'}