<jupyter_start><jupyter_text>Module 2.2: Working with CNNs in Keras (A Review)We turn to implementing a CNN in the Keras functional API. In this module we will pay attention to:1. Using the Keras functional API for defining models.2. Implementing dropout regularization.Those students who are comfortable with all these matters might consider skipping ahead.Note that we will not spend time tuning hyper-parameters: The purpose is to show how different techniques can be implemented in Keras, not to solve particular data science problems as optimally as possible. Obviously, most techniques include hyper-parameters that need to be tuned for optimal performance. We start by importing required libraries.<jupyter_code>import numpy as np

from sklearn.metrics import confusion_matrix,classification_report

from keras.datasets import cifar10
from keras.models import Sequential
from keras import Model
from keras.layers import Dense,Dropout,Flatten,Activation,Input
from keras.optimizers import Adam
from keras.layers.convolutional import Conv2D,MaxPooling2D
from keras.callbacks import EarlyStopping
from keras.utils import np_utils

import matplotlib.pyplot as plt<jupyter_output>Using TensorFlow backend.<jupyter_text>We will use the CIFAR10 dataset. This consists of small (32 x 32 pixel) color images of 10 different types of objects. It is included in the keras.datasets library.We load the images. These are already split into training and test cases. We need to normalize the pixel values to be between 0 and 1, and turn our integer labels into one-hot vectors - these are 1d-arrays of length the same as the number of classes, with zeros everywhere except the label specified, which is a 1. They are the probability that the image is of different classes.We also make a vector of class/label names for display purposes, as the label arrays contain only integers.<jupyter_code># Load images
(train_images, train_labels), (test_images, test_labels) = cifar10.load_data()

# Normalize pixel values to be between 0 and 1
train_images, test_images = train_images / 255.0, test_images / 255.0

# Make versions of the labels that are one-hot vectors
train_labels_array=np_utils.to_categorical(train_labels, 10)
test_labels_array=np_utils.to_categorical(test_labels, 10)

# Make vector of classnames
class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']
train_labels_array.shape<jupyter_output><empty_output><jupyter_text>Let's make a function to have a look at the images.<jupyter_code>def show_images(images,labels,class_names,random=True):
  plt.figure(figsize=(10,10))
  if random:
    indices=np.random.randint(0,images.shape[0],25)
  else:
    indices=np.array([i for i in range(25)])
  for i in range(25):
      plt.subplot(5,5,i+1)
      plt.xticks([])
      plt.yticks([])
      plt.grid(False)
      plt.imshow(images[indices[i]], cmap=plt.cm.binary)
      # The CIFAR labels happen to be arrays, 
      # which is why we need the extra index
      plt.xlabel(class_names[labels[indices[i]][0]])
  plt.show()<jupyter_output><empty_output><jupyter_text>Now we run it. We will see 25 random images from the dataset that we pass. If you set random=False you will see the first 25 images, the variety of which reassures us that the data is in a random order. (If this was a real world problem, such re-assurances would be insufficient, and we would shuffle the data.)<jupyter_code>show_images(train_images,train_labels,class_names,False)<jupyter_output><empty_output><jupyter_text>Now we create a function that will define the network architecture. Note that we introduce dropout layers for regularization purposes. We discussed these in the last module.For comparison, the code to specify the same network using the sequential approach is provided in a second function.<jupyter_code>def get_model():
  inputs = Input(shape=(32, 32, 3),name="Input")
  conv1 = Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3))(inputs)
  pool1 = MaxPooling2D((2, 2))(conv1)
  drop1 = Dropout(0.5)(pool1)
  conv2 = Conv2D(64, (3, 3), activation='relu')(drop1)
  pool2 = MaxPooling2D((2, 2))(conv2)
  drop2 = Dropout(0.5)(pool2)
  conv3 = Conv2D(64, (3, 3), activation='relu')(drop2)
  flat = Flatten()(conv3)
  dense1 = Dense(64, activation='relu')(flat)
  outputs = Dense(10, activation='softmax')(dense1)
  model = Model(inputs=inputs,outputs=outputs)
  return model

# For comparison, this is how we would use the sequential process
def get_model_seqential():
  model = Sequential()
  model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
  model.add(MaxPooling2D((2, 2)))
  model.add(Dropout(0.5))
  model.add(Conv2D(64, (3, 3), activation='relu'))
  model.add(MaxPooling2D((2, 2)))
  model.add(Dropout(0.5))
  model.add(Conv2D(64, (3, 3), activation='relu'))
  model.add(Flatten())
  model.add(Dense(64, activation='relu'))
  model.add(Dense(10, activation='softmax'))
  return model<jupyter_output><empty_output><jupyter_text>We will get our model.<jupyter_code>model=get_model()<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4267: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:148: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From /usr[...]<jupyter_text>Now we will define an optimizer and compile it. If you are unfamiliar with the different types of optimizers available in keras, I suggest you read the keras documentation [here](https://keras.io/optimizers/) and play around training the model with different alternatives.<jupyter_code>opt=Adam()<jupyter_output><empty_output><jupyter_text>And we compile our model with the optimizer ready for training. We use categorical crossentropy as our loss function as this is a good default choice for working with a multi-class categorical target variable (i.e. the image labels).<jupyter_code>model.compile(optimizer=opt,
              loss='categorical_crossentropy',
              metrics=['accuracy'])<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3576: The name tf.log is deprecated. Please use tf.math.log instead.<jupyter_text>Now we fit (train) the model. We will set the training to continue for 100 epochs, but use an early stopping callback which means it should terminate much quicker than this.<jupyter_code># Before calling fit, we create the Early Stopping callback. 
# We set it up to stop if improvement in the validation loss 
# does not occur over 10 epochs. When stopping occurs, the 
# weights associated with the best validation loss are restored.
earlyStopping = EarlyStopping(monitor="val_loss", 
                              patience=10,
                              verbose=1,
                              restore_best_weights=True)

# We need to use the one-hot vector version of the labels
# This shouldn't go through all 100 epoches, because of the
# early stopping, but can take some time.
history = model.fit(train_images, 
                    train_labels_array, 
                    epochs=100, 
                    shuffle=True, 
                    callbacks=[earlyStopping],
                    validation_split=.2)<jupyter_output>Train on 40000 samples, validate on 10000 samples
Epoch 1/100
40000/40000 [==============================] - 64s 2ms/step - loss: 1.3310 - acc: 0.5225 - val_loss: 1.2282 - val_acc: 0.5731
Epoch 2/100
40000/40000 [==============================] - 64s 2ms/step - loss: 1.2306 - acc: 0.5627 - val_loss: 1.1447 - val_acc: 0.5985
Epoch 3/100
40000/40000 [==============================] - 64s 2ms/step - loss: 1.1631 - acc: 0.5856 - val_loss: 1.0540 - val_acc: 0.6260
Epoch 4/100
40000/40000 [==============================] - 65s 2ms/step - loss: 1.1153 - acc: 0.6022 - val_loss: 1.0628 - val_acc: 0.6285
Epoch 5/100
40000/40000 [==============================] - 64s 2ms/step - loss: 1.0759 - acc: 0.6187 - val_loss: 1.0586 - val_acc: 0.6305
Epoch 6/100
40000/40000 [==============================] - 65s 2ms/step - loss: 1.0382 - acc: 0.6312 - val_loss: 0.9702 - val_acc: 0.6566
Epoch 7/100
40000/40000 [==============================] - 64s 2ms/step - loss: 1.0069 - acc: 0.6435 - val_loss: 0.9820 - [...]<jupyter_text>We will plot the training history to see a graphical representation of the training.<jupyter_code>def plot_training_history(history):
    plt.plot(history.history['acc'])
    plt.plot(history.history['val_acc'])
    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title('Model accuracy and loss')
    plt.xlabel('Epoch')
    plt.legend(['Accuracy','Validation Accuracy', 'Loss',
                'Validation Loss'], loc='upper right')
    plt.show()
plot_training_history(history)<jupyter_output><empty_output><jupyter_text>Finally, for fun lets see how our improved model performs on our test data. But remember that we have not spent any time or effort optimizing this model - for a real problem we would determine good values for the dropout regularization, as well as tune the architecture and optimizer.We make a function that will show the confusion matrix, and then run it.<jupyter_code>def test_model(model,x,y):
    y_pred = model.predict(x)
    y_pred = np.argmax(y_pred,axis=1)
    cm = confusion_matrix(y, y_pred)
    print("Confusion Matrix:")
    print(cm)
    print("Classification report:")
    print(classification_report(y, y_pred))
test_model(model,test_images,test_labels)<jupyter_output>Confusion Matrix:
[[695  15  51  10  32   9  12  11 130  35]
 [ 13 825   1   8  12   3  13   1  50  74]
 [ 55   2 543  50 137  78  72  26  30   7]
 [ 16   5  67 470  79 187  88  43  26  19]
 [  9   1  48  48 727  38  50  55  19   5]
 [  8   1  36 120  48 662  48  56  16   5]
 [  7   4  28  36  59  23 821   8  10   4]
 [ 10   4  23  30  67  61  10 776   4  15]
 [ 46  23   6  11   4   6   2   6 877  19]
 [ 24  62   7  10  13   5   8  13  42 816]]
Classification report:
              precision    recall  f1-score   support

           0       0.79      0.69      0.74      1000
           1       0.88      0.82      0.85      1000
           2       0.67      0.54      0.60      1000
           3       0.59      0.47      0.52      1000
           4       0.62      0.73      0.67      1000
           5       0.62      0.66      0.64      1000
           6       0.73      0.82      0.77      1000
           7       0.78      0.78      0.78      1000
           8       0.73      0.88      0.[...]
{'fraction_non_alphanumeric': 0.11222357652794707, 'fraction_numerical': 0.06738638342329793, 'mean_word_length': 3.613253012048193, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '13140', 'license': 'MIT', 'n_tokens_mistral': 4082, 'n_tokens_neox': 3586, 'n_words': 1172, 'path': 'Module 2.2_ CNNs.ipynb', 'repo_name': 'PacktPublishing/-Mastering-Keras', 'size': '11486'}