<jupyter_start><jupyter_text>3 Neural Networks in TensorFlow - Image Classification  Introduction - Classify images of clothing **This exercise trains a neural network model to classify images of clothing, like sneakers and shirts.****It uses tf.keras, a high-level API to build and train models in TensorFlow.**  Import libraries<jupyter_code># TensorFlow and tf.keras
import tensorflow as tf

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt

print(tf.__version__)<jupyter_output>2.3.0<jupyter_text>Import images from Fashion MNIST  **The classic MNIST dataset is often used as the "Hello, World" of machine learning programs for computer vision. [Fashion MNIST](https://github.com/zalandoresearch/fashion-mnist) is intended to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms.****Fashion MNIST is a slightly more challenging problem than the classic MNIST. Instead of classifying the handwritten digits (0, 1, 2, etc.), it focuses on clothings.****In the Fashion MNIST dataset, it contains 70,000 grayscale images in 10 categories.**            Label        Description                0        T-shirt/top                1        Trouser                2        Pullover                3        Dress                4        Coat                5        Sandal                6        Shirt                7        Sneaker                8        Bag                9        Ankle boot     **Each example is a 28x28 grayscale image. Here's an example of how the data looks (each class takes three-rows).**                        Reference: https://github.com/zalandoresearch/fashion-mnist **The Fashion MNIST data can be imported and loaded directly from TensorFlow.****The dataset contains 60,000 28x28 grayscale images of 10 fashion categories, along with a test set of 10,000 images.****tf.keras.datasets.fashion_mnist.load_data() returns four numpy arrays:**>**train_images is a numpy array of grayscale image data with shapes (60000, 28, 28), containing the training data.**>**train_labels is a numpy array of labels (integers in range 0-9) with shape (60000,) for the training image.**>**test_images is a numpy array of grayscale image data with shapes (10000, 28, 28), containing the test data.**>**test_labels is a numpy array of labels (integers in range 0-9) with shape (10000,) for the test image.**<jupyter_code>fashion_mnist = tf.keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()<jupyter_output><empty_output><jupyter_text>**The images are stored in 28x28 numpy arrays, with pixel values ranging from 0 to 255. The labels are an array of integers, ranging from 0 to 9. Their corresponding class names can be stored in a list.**<jupyter_code>class_names = ['T-shirt/top', 
               'Trouser', 
               'Pullover', 
               'Dress', 
               'Coat', 
               'Sandal', 
               'Shirt', 
               'Sneaker', 
               'Bag', 
               'Ankle boot']<jupyter_output><empty_output><jupyter_text>Explore the data  **Shapes of training and test images**<jupyter_code>print("The shape of training images is:", train_images.shape)
print("The shape of training labels is:", train_labels.shape)
print("The shape of test images is:", test_images.shape)
print("The shape of test labels is:", test_labels.shape)<jupyter_output>The shape of test images is: (10000, 28, 28)
The shape of test labels is: (10000,)<jupyter_text>**Label values**<jupyter_code>print("The label values range from:", np.min(train_labels), "to" ,np.max(train_labels))<jupyter_output>The label values range from: 0 to 9<jupyter_text>**Pixel values**<jupyter_code>print("The pixel values range from:", np.min(train_images), "to" ,np.max(train_images))

plt.figure(figsize=(17,8))
for i in range(32):
    plt.subplot(4,8,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.imshow(train_images[i])
    plt.xlabel(class_names[train_labels[i]])
plt.show()<jupyter_output>The pixel values range from: 0 to 255<jupyter_text>Preprocess the data - Normalization  &nbsp;!&nbsp;**Data normalization is used in machine learning to make model training less sensitive to the scale of features. This allows model to converge to better weights and, in turn, leads to a more accurate model.****In practice, normalization refers to rescaling real-valued numeric attributes into a range of 0 to 1. Data will be normalized before feeding them to the neural network model.****It's important that the training set and testing set be preprocessed in the same way. In this case, the pixel values of both training and test images are divided by 255.**<jupyter_code>train_images_normal = train_images / 255.0

test_images_normal = test_images / 255.0
print("The pixel values range from:", np.min(train_images_normal ), "to" ,np.max(train_images_normal))

plt.figure(figsize=(17,8))
for i in range(32):
    plt.subplot(4,8,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.imshow(train_images_normal[i], cmap=plt.cm.binary)
    plt.xlabel(class_names[train_labels[i]])
plt.show()<jupyter_output>The pixel values range from: 0.0 to 1.0<jupyter_text>Build the model   *1) Setup and configure a Sequential model*<jupyter_code>model = tf.keras.Sequential()
model.add(tf.keras.layers.Flatten(input_shape=(28,28)))
model.add(tf.keras.layers.Dense(128, activation ='relu'))
model.add(tf.keras.layers.Dense(10, activation ='softmax'))
model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten (Flatten)            (None, 784)               0         
_________________________________________________________________
dense (Dense)                (None, 128)               100480    
_________________________________________________________________
dense_1 (Dense)              (None, 10)                1290      
=================================================================
Total params: 101,770
Trainable params: 101,770
Non-trainable params: 0
_________________________________________________________________<jupyter_text>**1st layer:**> **tf.keras.layers.Flatten transforms the format of the images from a two-dimensional array (28 by 28 pixels) to a one-dimensional array (28x28 = 784 pixels).**> **This layer has no parameters to learn, which only serves to reformat the data by unstacking rows of pixels in the image and lining them up.****2nd and 3rd layers:**> **After the pixels are flattened, the network consists of a sequence of two tf.keras.layers.Dense layers. These layers are densely connected or fully connected.**> **The 2nd layer has 128 nodes (or neurons).**> **The 3rd layer has 10 nodes (or neurons) which returns a logits array with length of 10. Each node contains a score that indicates the current image belongs to one of the 10 classes.**   *2) Compile the model*<jupyter_code>model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>**SparseCategoricalCrossentropy and CategoricalCrossentropy both compute categorical cross-entropy. The only difference is in how the targets/labels should be encoded.**> **For one hot representation, CatergoricalCrossentropy can be used.**> **For labels as integers (i.e. 0, 1, 2), SparseCategoricalCrossentropy can be used.** **from_logits argument informs the loss function whether the output is expected to be a logits tensor (i.e. normalized).**>**By default, from_logits is False  which assumes that output encodes a probability distribution, such as outputs from softmax and sigmoid activations.**>**If from_logits is True, this assumes that output is a logits. In this case, softmax or sigmoid activations will automatically applied on the output values by the loss function. Hence, output layer doesn't need any probabilistic activation function.** &nbsp;!&nbsp; **Remark: If from_logits is True, no activation function is needed in the output layer, e.g.**   Train the model   *1) Feed the model*<jupyter_code>model.fit(train_images_normal, train_labels, epochs=10)<jupyter_output>Epoch 1/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.5026 - accuracy: 0.8217
Epoch 2/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3752 - accuracy: 0.8655
Epoch 3/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3368 - accuracy: 0.8773
Epoch 4/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3129 - accuracy: 0.8856
Epoch 5/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2948 - accuracy: 0.8914: 0s - loss: 0.2947 - accuracy: 0.89
Epoch 6/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2801 - accuracy: 0.8959
Epoch 7/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2695 - accuracy: 0.8998
Epoch 8/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2564 - accuracy: 0.9051
Epoch 9/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2472 - accuracy: 0.9075
Epoch 10/10
1875/1875 [=======================[...]<jupyter_text>**To start training, model.fit() method can be used to "fit" the model with training data. In this example, the training data is in the train_images_normal and train_labels arrays.*****Training accuracy***> **As the model trains, the loss and accuracy metrics are improving. After 10 epochs, this model reaches an accuracy of about 0.91 (91%) on training data.**  *2) Evaluate the model*<jupyter_code>test_loss, test_acc = model.evaluate(test_images,  test_labels, verbose=2)

print('Test accuracy:', round(test_acc*100), '%')<jupyter_output>313/313 - 0s - loss: 56.6535 - accuracy: 0.8622
Test accuracy: 86 %<jupyter_text>***Test accuracy***> **It turns out that the accuracy on the test dataset is a little less than the accuracy on the training dataset. This gap between training accuracy and test accuracy represents overfitting.**> **Overfitting happens when a machine learning model performs worse on new, previously unseen inputs than it does on the training data. An overfitted model "memorizes" the noise and details in the training dataset to a point where it negatively impacts the performance of the model on the new data. In other words, its generalization ability is poor.**    Reference:*   [Demonstrate overfitting](https://www.tensorflow.org/tutorials/keras/overfit_and_underfitdemonstrate_overfitting)*   [Strategies to prevent overfitting](https://www.tensorflow.org/tutorials/keras/overfit_and_underfitstrategies_to_prevent_overfitting)  *3) Make predictions* **Images can be fed into the trained network to make predictions.**<jupyter_code>predictions = model.predict(test_images_normal)<jupyter_output><empty_output><jupyter_text>**A prediction is an array of 10 numbers. They represent the model's confidence that the image corresponds to each of the 10 different articles of clothing. The highest confidence value indicates the prediction.** **Let's take a look at the first prediction.**<jupyter_code>predictions[0]<jupyter_output><empty_output><jupyter_text>**The highest confidence is the last class in the array. Let's find out which class it belongs to.**<jupyter_code>max_confidence = np.argmax(predictions[0])

print("The highest confidence is:", max_confidence)
print("The predicted class is:", class_names[max_confidence])<jupyter_output>The highest confidence is: 9
The predicted class is: Ankle boot<jupyter_text>**The model is most confident that this image is an ankle boot. Let's examine the test label and see if it predicts correctly.**<jupyter_code>print("The test label is:", test_labels[0])
print("The test class is:", class_names[test_labels[0]])<jupyter_output>The test label is: 9
The test class is: Ankle boot<jupyter_text>**The model predicts correctly.**  *4) Verify and visualize predictions* **With trained network, it can be used to make predictions.****Let's predict 16 images. Correct prediction labels are blue and incorrect prediction labels are red. The number indicates the precentage (i.e. confidence) for the predicted label.**<jupyter_code>plt.figure(figsize=(17,8))

for i in range(16):
    plt.subplot(4,8,2*i+1)
    plt.xticks([])
    plt.yticks([])
    if np.argmax(predictions[i]) == test_labels[i]:
        color = 'blue'
    else:
        color = 'red'
    plt.imshow(test_images[i], cmap=plt.cm.binary)
    plt.xlabel("{} {:2.0f}% ({})".format(class_names[np.argmax(predictions[i])],
                                100*np.max(predictions[i]),
                                class_names[test_labels[i]]),
                                color=color)
    plt.subplot(4,8,2*i+2)
    plt.xticks(range(10))
    plt.yticks([])
    thisplot = plt.bar(range(10), predictions[i], color="grey")
    plt.ylim([0, 1])  

    thisplot[np.argmax(predictions[i])].set_color('red')
    thisplot[test_labels[i]].set_color('blue')
    
plt.show()<jupyter_output><empty_output><jupyter_text>&nbsp;!&nbsp; **The model can be wrong even when it has high confidence about its prediction.**  Use the trained model  **Once the model is trained, images can be fed into the model for prediction.**<jupyter_code># Grab an image from the test dataset.
img = test_images[50]
print(img.shape)<jupyter_output>(28, 28)<jupyter_text>**tf.keras models are optimized to make predictions on a batch, or collection, of examples at once. Therefore, even this is a single image prediction, image needs to be add into a list and make it like a batch.****np.expand_dims function can help insert a new axis that will appear at the axis position in the expanded array shape.**<jupyter_code># Add the image to a batch where it's the only member.
img = (np.expand_dims(img,0))

print(img.shape)<jupyter_output>(1, 28, 28)<jupyter_text>**Then, images can be fed into the model for prediction.**<jupyter_code>predictions_img = model.predict(img)

max_confidence = np.argmax(predictions_img[0])

print("The highest confidence is:", max_confidence)
print("The predicted class is:", class_names[max_confidence])<jupyter_output>The highest confidence is: 4
The predicted class is: Coat<jupyter_text>**Let's visualize the results.**<jupyter_code>plt.figure(figsize=(12,4))

i=50
plt.subplot(1,2,1)
plt.xticks([])
plt.yticks([])
if np.argmax(predictions[i]) == test_labels[i]:
    color = 'blue'
else:
    color = 'red'
plt.imshow(test_images[i], cmap=plt.cm.binary)
plt.xlabel("{} {:2.0f}% ({})".format(class_names[np.argmax(predictions[i])],
                            100*np.max(predictions[i]),
                            class_names[test_labels[i]]),
                            color=color)
plt.subplot(1,2,2)
plt.xticks(range(10), class_names, rotation=45)
plt.yticks([])
thisplot = plt.bar(range(10), predictions[i], color="grey")
plt.ylim([0, 1])  

thisplot[np.argmax(predictions[i])].set_color('red')
thisplot[test_labels[i]].set_color('blue')
    
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1309100487290926, 'fraction_numerical': 0.03621756881338074, 'mean_word_length': 4.302723463687151, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 84, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '574851', 'license': 'MIT', 'n_tokens_mistral': 4879, 'n_tokens_neox': 4374, 'n_words': 1522, 'path': '3 - Neural Networks in TensorFlow - Exercise Image Classification.ipynb', 'repo_name': 'andpyxa/Deep-Learning-in-TensorFlow-Specialization', 'size': '15186'}