<jupyter_start><jupyter_text>Dependencies<jupyter_code># Dependencies to Visualize the model
%matplotlib inline
from IPython.display import Image, SVG
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(0)
# Filepaths, numpy, and Tensorflow
import os
import numpy as np
import tensorflow as tf
# Sklearn scaling
from sklearn.preprocessing import MinMaxScaler<jupyter_output><empty_output><jupyter_text>Keras Specific Dependencies<jupyter_code># Keras
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.layers import Dense
from tensorflow.keras.datasets import mnist<jupyter_output><empty_output><jupyter_text>Loading and Preprocessing our Data  Load the MNIST Handwriting Dataset from Keras<jupyter_code>(X_train, y_train), (X_test, y_test) = mnist.load_data()
print("Training Data Info")
print("Training Data Shape:", X_train.shape)
print("Training Data Labels Shape:", y_train.shape)<jupyter_output>Training Data Info
Training Data Shape: (60000, 28, 28)
Training Data Labels Shape: (60000,)<jupyter_text>Plot the first digit<jupyter_code># Plot the first image from the dataset
plt.imshow(X_train[0,:,:], cmap=plt.cm.Greys)<jupyter_output><empty_output><jupyter_text>Each Image is a 28x28 Pixel greyscale image with values from 0 to 255<jupyter_code># Our image is an array of pixels ranging from 0 to 255
X_train[0, :, :]<jupyter_output><empty_output><jupyter_text>For training a model, we want to flatten our data into rows of 1D image arrays<jupyter_code># We want to flatten our image of 28x28 pixels to a 1D array of 784 pixels
ndims = X_train.shape[1] * X_train.shape[2]
X_train = X_train.reshape(X_train.shape[0], ndims)
X_test = X_test.reshape(X_test.shape[0], ndims)
print("Training Shape:", X_train.shape)
print("Testing Shape:", X_test.shape)<jupyter_output>Training Shape: (60000, 784)
Testing Shape: (10000, 784)<jupyter_text>Scaling and NormalizationWe use Sklearn's MinMaxScaler to normalize our data between 0 and 1<jupyter_code># Next, we normalize our training data to be between 0 and 1
scaler = MinMaxScaler().fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# Alternative way to normalize this dataset since we know that the max pixel value is 255
# X_train = X_train.astype("float32")
# X_test = X_test.astype("float32")
# X_train /= 255.0
# X_test /= 255.0<jupyter_output><empty_output><jupyter_text>One-Hot EncodingWe need to one-hot encode our integer labels using the `to_categorical` helper function<jupyter_code># Our Training and Testing labels are integer encoded from 0 to 9
y_train[:20]
# We need to convert our target labels (expected values) to categorical data
num_classes = 10
y_train = to_categorical(y_train, num_classes)
y_test = to_categorical(y_test, num_classes)
# Original label of `5` is one-hot encoded as `0000010000`
y_train[0]<jupyter_output><empty_output><jupyter_text>Building our ModelIn this example, we are going to build a Deep Multi-Layer Perceptron model with 2 hidden layers.  Our first step is to create an empty sequential model<jupyter_code># Create an empty sequential model
model = Sequential()<jupyter_output><empty_output><jupyter_text>Next, we add our first hidden layerIn the first hidden layer, we must also specify the dimension of our input layer. This will simply be the number of elements (pixels) in each image.<jupyter_code># Add the first layer where the input dimensions are the 784 pixel values
# We can also choose our activation function. `relu` is a common
model.add(Dense(100, activation='relu', input_dim=X_train.shape[1]))<jupyter_output><empty_output><jupyter_text>We then add a second hidden layer with 100 densely connected nodesA dense layer is when every node from the previous layer is connected to each node in the current layer.<jupyter_code># Add a second hidden layer
model.add(Dense(100, activation='relu'))<jupyter_output><empty_output><jupyter_text>Our final output layer uses a `softmax` activation function for logistic regression.We also need to specify the number of output classes. In this case, the number of digits that we wish to classify.<jupyter_code># Add our final output layer where the number of nodes 
# corresponds to the number of y labels
model.add(Dense(num_classes, activation='softmax'))<jupyter_output><empty_output><jupyter_text>Model Summary<jupyter_code># We can summarize our model
model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 100)               78500     
_________________________________________________________________
dense_1 (Dense)              (None, 100)               10100     
_________________________________________________________________
dense_2 (Dense)              (None, 10)                1010      
=================================================================
Total params: 89,610
Trainable params: 89,610
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Compile and Train our ModelNow that we have our model architecture defined, we must compile the model using a loss function and optimizer. We can also specify additional training metrics such as accuracy.<jupyter_code># Use categorical crossentropy for categorical data and mean squared error for regression
# Hint: your output layer in this example is using software for logistic regression (categorical)
# If your output layer activation was `linear` then you may want to use `mse` for loss
model.compile(optimizer='adam', 
              loss='categorical_crossentropy', 
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Finally, we train our model using our training data Training consists of updating our weights using our optimizer and loss function. In this example, we choose 10 iterations (loops) of training that are called epochs.We also choose to shuffle our training data and increase the detail printed out during each training cycle.<jupyter_code># Fit (train) the model
model.fit(
    X_train,
    y_train,
    epochs=10,
    shuffle=True,
    verbose=2
)<jupyter_output>Epoch 1/10
 - 10s - loss: 0.2431 - acc: 0.9287
Epoch 2/10
 - 9s - loss: 0.1052 - acc: 0.9684
Epoch 3/10
 - 10s - loss: 0.0750 - acc: 0.9764
Epoch 4/10
 - 9s - loss: 0.0569 - acc: 0.9827
Epoch 5/10
 - 9s - loss: 0.0466 - acc: 0.9846
Epoch 6/10
 - 9s - loss: 0.0380 - acc: 0.9879
Epoch 7/10
 - 9s - loss: 0.0327 - acc: 0.9890
Epoch 8/10
 - 8s - loss: 0.0274 - acc: 0.9910
Epoch 9/10
 - 8s - loss: 0.0214 - acc: 0.9933
Epoch 10/10
 - 8s - loss: 0.0211 - acc: 0.9930<jupyter_text>Saving and Loading modelsWe can save our trained models using the HDF5 binary format with the extension `.h5`<jupyter_code># Save the model
model.save("mnist_trained.h5")
# Load the model
from tensorflow.keras.models import load_model
model = load_model("mnist_trained.h5")<jupyter_output><empty_output><jupyter_text>Evaluating the ModelWe use our testing data to validate our model. This is how we determine the validity of our model (i.e. the ability to predict new and previously unseen data points)<jupyter_code># Evaluate the model using the training data 
model_loss, model_accuracy = model.evaluate(X_test, y_test, verbose=2)
print(f"Loss: {model_loss}, Accuracy: {model_accuracy}")<jupyter_output>Loss: 0.0957128526122302, Accuracy: 0.9777<jupyter_text>Making PredictionsWe can use our trained model to make predictions using `model.predict`<jupyter_code># Grab just one data point to test with
test = np.expand_dims(X_train[0], axis=0)
test.shape
plt.imshow(scaler.inverse_transform(test).reshape(28, 28), cmap=plt.cm.Greys)
# Make a prediction. The result should be 0000010000000 for a 5
model.predict(test).round()
# Grab just one data point to test with
test = np.expand_dims(X_train[2], axis=0)
test.shape
plt.imshow(scaler.inverse_transform(test).reshape(28, 28), cmap=plt.cm.Greys)
# Make a prediction. The resulting class should match the digit
print(f"One-Hot-Encoded Prediction: {model.predict(test).round()}")
print(f"Predicted class: {model.predict_classes(test)}")<jupyter_output>One-Hot-Encoded Prediction: [[0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]]
Predicted class: [4]<jupyter_text>Import a Custom Image<jupyter_code>filepath = "../Images/test8.png"
from tensorflow.keras.preprocessing import image
image_size = (28, 28)
im = image.load_img(filepath, target_size=image_size, color_mode="grayscale")
im
# Convert the image to a numpy array 
from tensorflow.keras.preprocessing.image import img_to_array
image = img_to_array(im)
image.shape
# Scale the image pixels by 255 (or use a scaler from sklearn here)
image /= 255

# Flatten into a 1x28*28 array 
img = image.flatten().reshape(-1, 28*28)
img.shape
plt.imshow(img.reshape(28, 28), cmap=plt.cm.Greys)
# Invert the pixel values to match the original data
img = 1 - img
plt.imshow(img.reshape(28, 28), cmap=plt.cm.Greys)
# Make predictions
model.predict_classes(img)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10031586973096612, 'fraction_numerical': 0.04204335039756018, 'mean_word_length': 5.310652920962199, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '370994', 'license': 'ADSL', 'n_tokens_mistral': 3207, 'n_tokens_neox': 2853, 'n_words': 973, 'path': '01-Lesson-Plans/23-Project-4/1/Activities/01-Evr_MNIST/Solved/MNIST.ipynb', 'repo_name': 'anirudhmungre/sneaky-lessons', 'size': '9181'}