<jupyter_start><jupyter_text>Setup<jupyter_code>! shred -u setup_google_colab.py
! wget https://raw.githubusercontent.com/hse-aml/intro-to-dl/master/setup_google_colab.py -O setup_google_colab.py
  
import setup_google_colab
setup_google_colab.setup_week3()
import grading_utils
import grading

grader = grading.Grader(
    assignment_key="<KEY>", 
    all_parts=["7W4tu", "nQOsg", "96eco"])

# token expires every 30 min
COURSERA_TOKEN = '<PASSWORD>'
COURSERA_EMAIL = '<EMAIL>'<jupyter_output><empty_output><jupyter_text>CNN on CIFAR-10CIFAR-10 dataset contains 32x32 color images from 10 classes: __airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck__:<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import display
# use preloaded keras datasets and models
! mkdir -p ~/.keras/datasets
! mkdir -p ~/.keras/models
! ln -s $(realpath ../readonly/keras/datasets/*) ~/.keras/datasets/
! ln -s $(realpath ../readonly/keras/models/*) ~/.keras/models/<jupyter_output>ln: failed to create symbolic link '/root/.keras/datasets/cifar-10-batches-py.tar.gz': File exists
ln: failed to create symbolic link '/root/.keras/datasets/mnist.npz': File exists
ln: failed to create symbolic link '/root/.keras/models/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5': File exists<jupyter_text>Dataset<jupyter_code>from keras.datasets import cifar10

(x_train, y_train), (x_test, y_test) = cifar10.load_data()
print(x_train.shape)
print(y_train.shape)
print('')
print(x_test.shape)
print(y_test.shape)
# Output classes
NUM_CLASSES = 10
cifar10_classes = [
    "airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "ship", "truck"]
# show random images from trainning set
cols = 8
rows = 2
fig = plt.figure(figsize=(2 * cols - 1, 2.5 * rows - 1))
for i in range(cols):
    for j in range(rows):
        random_index = np.random.randint(0, len(y_train))
        ax = fig.add_subplot(rows, cols, i * rows + j + 1)
        ax.grid('off')
        ax.axis('off')
        ax.imshow(x_train[random_index, :])
        ax.set_title(cifar10_classes[y_train[random_index, 0]])
plt.show()<jupyter_output><empty_output><jupyter_text>Data Engineering We need to normalize inputs like this: $$x_{norm} = \frac{x}{255} - 0.5$$<jupyter_code># normalize inputs
x_train2 = x_train / 255.0 - 0.5 ### YOUR CODE HERE
x_test2 = x_test / 255.0 - 0.5 ### YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>We need to convert class labels to one-hot encoded vectors. Use __keras.utils.to_categorical__.<jupyter_code># convert class labels to one-hot encoded, should have shape (?, NUM_CLASSES)
y_train2 = np.eye(NUM_CLASSES)[y_train[:,0]] ### YOUR CODE HERE
y_test2 = np.eye(NUM_CLASSES)[y_test[:,0]] ### YOUR CODE HERE

print(x_train2.shape, y_train2.shape)
print(x_test2.shape, y_test2.shape)<jupyter_output>(50000, 32, 32, 3) (50000, 10)
(10000, 32, 32, 3) (10000, 10)<jupyter_text>ModelConvolutional networks are built from several types of layers:- [Conv2D](https://keras.io/layers/convolutional/conv2d) - performs convolution:    - **filters**: number of output channels;     - **kernel_size**: an integer or tuple/list of 2 integers, specifying the width and height of the 2D convolution window;    - **padding**: padding="same" adds zero padding to the input, so that the output has the same width and height, padding='valid' performs convolution only in locations where kernel and the input fully overlap;    - **activation**: "relu", "tanh", etc.    - **input_shape**: shape of input.- [MaxPooling2D](https://keras.io/layers/pooling/maxpooling2d) - performs 2D max pooling.- [Flatten](https://keras.io/layers/core/flatten) - flattens the input, does not affect the batch size.- [Dense](https://keras.io/layers/core/dense) - fully-connected layer.- [Activation](https://keras.io/layers/core/activation) - applies an activation function.- [LeakyReLU](https://keras.io/layers/advanced-activations/leakyrelu) - applies leaky relu activation.- [Dropout](https://keras.io/layers/core/dropout) - applies dropout. Model SuggestionYou need to define a model which takes __(None, 32, 32, 3)__ input and predicts __(None, 10)__ output with probabilities for all classes. __None__ in shapes stands for batch dimension.Simple feed-forward networks in Keras can be defined in the following way:```pythonmodel = Sequential()   start feed-forward model definitionmodel.add(Conv2D(..., input_shape=(32, 32, 3)))   first layer needs to define "input_shape"...   here comes a bunch of convolutional, pooling and dropout layersmodel.add(Dense(NUM_CLASSES))   the last layer with neuron for each classmodel.add(Activation("softmax"))   output probabilities```Stack __4__ convolutional layers with kernel size __(3, 3)__ with growing number of filters __(16, 32, 32, 64)__, use "same" padding.Add __2x2__ pooling layer after every 2 convolutional layers (conv-conv-pool scheme).Use __LeakyReLU__ activation with recommended parameter __0.1__ for all layers that need it (after convolutional and dense layers):```pythonmodel.add(LeakyReLU(0.1))```Add a dense layer with __256__ neurons and a second dense layer with __10__ neurons for classes. Remember to use __Flatten__ layer before first dense layer to reshape input volume into a flat vector!Add __Dropout__ after every pooling layer (__0.25__) and between dense layers (__0.5__).<jupyter_code>def make_model():
    """
    Define your model architecture here.
    Returns `Sequential` model.
    """
    from keras.models import Sequential
    from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Activation, Dropout
    from keras.layers.advanced_activations import LeakyReLU
    
    model = Sequential()

    ### YOUR CODE HERE
    model.add(Conv2D(filters=16, kernel_size=(3, 3), input_shape=(32,32,3), padding="same"))
    model.add(LeakyReLU(0.1))
    
    model.add(Conv2D(filters=32, kernel_size=(3, 3), padding="same"))
    model.add(LeakyReLU(0.1))
    
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    
    model.add(Conv2D(filters=32, kernel_size=(3, 3), padding="same"))
    model.add(LeakyReLU(0.1))
    
    model.add(Conv2D(filters=64, kernel_size=(3, 3), padding="same"))
    model.add(LeakyReLU(0.1))
    
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    
    model.add(Flatten())
    model.add(Dense(256))
    
    model.add(LeakyReLU(0.1))
    model.add(Dropout(0.5))
    model.add(Dense(NUM_CLASSES))
    model.add(Activation("softmax"))

    return model
from keras import backend as K

K.clear_session()
model = make_model()

model.summary()
## GRADED PART, DO NOT CHANGE!

# Number of model parameters
grader.set_answer("7W4tu", grading_utils.model_total_params(model))

# you can make submission with answers so far to check yourself at this stage
grader.submit(COURSERA_EMAIL, COURSERA_TOKEN)<jupyter_output>Submitted to Coursera platform. See results on assignment page!<jupyter_text>Train Model<jupyter_code>import keras

INIT_LR = 5e-3  # initial learning rate
BATCH_SIZE = 32
EPOCHS = 30

# clear default graph
K.clear_session()  
# don't call K.set_learning_phase() !!! (otherwise will enable dropout in train/test simultaneously)

model = make_model()

# prepare model for fitting (loss, optimizer, etc)
model.compile(
    loss='categorical_crossentropy',                    # we train 10-way classification
    optimizer=keras.optimizers.adamax(lr=INIT_LR),      # for SGD
    metrics=['accuracy']                                # report accuracy during training
)

# scheduler of learning rate (decay with epochs)
def lr_scheduler(epoch):
    return INIT_LR * 0.9 ** epoch

# callback for printing of actual learning rate used by optimizer
class LrHistory(keras.callbacks.Callback):
    def on_epoch_begin(self, epoch, logs={}):
        print("Learning rate:", K.get_value(model.optimizer.lr))

# fit model
history = model.fit(
    x_train2, y_train2,
    epochs=EPOCHS,
    callbacks=[keras.callbacks.LearningRateScheduler(lr_scheduler), LrHistory()],
    validation_data=(x_test2, y_test2),
    shuffle=True,
)

historydf = pd.DataFrame(history.history, index=history.epoch)
historydf.plot();
# save weights to file
model.save_weights("weights.h5")<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code># load weights from file
model.load_weights("weights.h5")
# make test predictions
y_pred_test = model.predict_proba(x_test2)
y_pred_test_classes = np.argmax(y_pred_test, axis=1)
y_pred_test_max_probas = np.max(y_pred_test, axis=1)
# confusion matrix and accuracy
from sklearn.metrics import confusion_matrix, accuracy_score
plt.figure(figsize=(7, 6))
plt.title('Confusion matrix', fontsize=16)
plt.imshow(confusion_matrix(y_test, y_pred_test_classes))
plt.xticks(np.arange(10), cifar10_classes, rotation=45, fontsize=12)
plt.yticks(np.arange(10), cifar10_classes, fontsize=12)
plt.colorbar()
plt.show()
print("Test accuracy:", accuracy_score(y_test, y_pred_test_classes))
# inspect preditions
cols = 8
rows = 2
fig = plt.figure(figsize=(2 * cols - 1, 3 * rows - 1))
for i in range(cols):
    for j in range(rows):
        random_index = np.random.randint(0, len(y_test))
        ax = fig.add_subplot(rows, cols, i * rows + j + 1)
        ax.grid('off')
        ax.axis('off')
        ax.imshow(x_test[random_index, :])
        pred_label = cifar10_classes[y_pred_test_classes[random_index]]
        pred_proba = y_pred_test_max_probas[random_index]
        true_label = cifar10_classes[y_test[random_index, 0]]
        ax.set_title("pred: {}\nscore: {:.3}\ntrue: {}".format(
               pred_label, pred_proba, true_label
        ))
plt.show()
## GRADED PART, DO NOT CHANGE!

# Accuracy on validation data
grader.set_answer("nQOsg", accuracy_score(y_test, y_pred_test_classes))

# you can make submission with answers so far to check yourself at this stage
grader.submit(COURSERA_EMAIL, COURSERA_TOKEN)<jupyter_output>Submitted to Coursera platform. See results on assignment page!<jupyter_text>Visualize maximum stimuliWe want to find input images that provide maximum activations for particular layers of our network. We will find those maximum stimuli via gradient ascent in image space.For that task we load our model weights, calculate the layer output gradient with respect to image input and shift input image in that direction.<jupyter_code>K.clear_session()        # clear default graph
K.set_learning_phase(0)  # disable dropout

model = make_model()
model.load_weights("weights.h5")  # that were saved after model.fit
# all weights we have
model.summary()
def find_maximum_stimuli(layer_name, is_conv, filter_index, model, iterations=20, step=1., verbose=True):
    
    def image_values_to_rgb(x):
        # normalize x: center on 0 (np.mean(x_train2)), ensure std is 0.25 (np.std(x_train2))
        # so that it looks like a normalized image input for our network
        x = (x-np.mean(x_train2)) / np.std(x_train2) ### YOUR CODE HERE

        # do reverse normalization to RGB values: x = (x_norm + 0.5) * 255
        x = (x + 0.5) * 255 ### YOUR CODE HERE
    
        # clip values to [0, 255] and convert to bytes
        x = np.clip(x, 0, 255).astype('uint8')
        return x

    # this is the placeholder for the input image
    input_img = model.input
    img_width, img_height = input_img.shape.as_list()[1:3]
    
    # find the layer output by name
    layer_output = list(filter(lambda x: x.name == layer_name, model.layers))[0].output

    # we build a loss function that maximizes the activation
    # of the filter_index filter of the layer considered
    if is_conv:
        # mean over feature map values for convolutional layer
        loss = K.mean(layer_output[:, :, :, filter_index])
    else:
        loss = K.mean(layer_output[:, filter_index])

    # we compute the gradient of the loss wrt input image
    grads = K.gradients(loss, input_img)[0]  # [0] because of the batch dimension!

    # normalization trick: we normalize the gradient
    grads = grads / (K.sqrt(K.sum(K.square(grads))) + 1e-10)

    # this function returns the loss and grads given the input picture
    iterate = K.function([input_img], [loss, grads])

    # we start from a gray image with some random noise
    input_img_data = np.random.random((1, img_width, img_height, 3))
    input_img_data = (input_img_data - 0.5) * (0.1 if is_conv else 0.001)

    # we run gradient ascent
    for i in range(iterations):
        loss_value, grads_value = iterate([input_img_data])
        input_img_data += grads_value * step
        if verbose:
            print('Current loss value:', loss_value)

    # decode the resulting input image
    img = image_values_to_rgb(input_img_data[0])
    
    return img, loss_value
# sample maximum stimuli
def plot_filters_stimuli(layer_name, is_conv, model, iterations=20, step=1., verbose=False):
    cols = 8
    rows = 2
    filter_index = 0
    max_filter_index = list(filter(lambda x: x.name == layer_name, model.layers))[0].output.shape.as_list()[-1] - 1
    fig = plt.figure(figsize=(2 * cols - 1, 3 * rows - 1))
    for i in range(cols):
        for j in range(rows):
            if filter_index <= max_filter_index:
                ax = fig.add_subplot(rows, cols, i * rows + j + 1)
                ax.grid('off')
                ax.axis('off')
                loss = -1e20
                while loss < 0 and filter_index <= max_filter_index:
                    stimuli, loss = find_maximum_stimuli(layer_name, is_conv, filter_index, model,
                                                         iterations, step, verbose=verbose)
                    filter_index += 1
                if loss > 0:
                    ax.imshow(stimuli)
                    ax.set_title("Filter #{}".format(filter_index))
    plt.show()
from keras.layers.advanced_activations import LeakyReLU
from keras.layers import Conv2D, Dense

# maximum stimuli for convolutional neurons
conv_activation_layers = []
for layer in model.layers:
    if isinstance(layer, LeakyReLU):
        prev_layer = layer.inbound_nodes[0].inbound_layers[0]
        if isinstance(prev_layer, Conv2D):
            conv_activation_layers.append(layer)

for layer in conv_activation_layers:
    print(layer.name)
    plot_filters_stimuli(layer_name=layer.name, is_conv=True, model=model)
# maximum stimuli for last dense layer
last_dense_layer = list(filter(lambda x: isinstance(x, Dense), model.layers))[-1]
plot_filters_stimuli(
    layer_name=last_dense_layer.name, is_conv=False, 
    iterations=200, step=0.1, model=model)
def maximum_stimuli_test_for_grader():
    layer = list(filter(lambda x: isinstance(x, Dense), model.layers))[-1]
    output_index = 7
    stimuli, loss = find_maximum_stimuli(
        layer_name=layer.name, 
        is_conv=False, 
        filter_index=output_index,
        model=model,
        verbose=False
    )
    return model.predict_proba(stimuli[np.newaxis, :])[0, output_index]
## GRADED PART, DO NOT CHANGE!

# Maximum stimuli test
grader.set_answer("96eco", maximum_stimuli_test_for_grader())

# you can make submission with answers so far to check yourself at this stage
grader.submit(COURSERA_EMAIL, COURSERA_TOKEN)<jupyter_output>1/1 [==============================] - 0s
Submitted to Coursera platform. See results on assignment page!
{'fraction_non_alphanumeric': 0.11454865870373997, 'fraction_numerical': 0.02258338228575158, 'mean_word_length': 3.9827642276422766, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 36, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '259010', 'license': 'Unlicense', 'n_tokens_mistral': 5346, 'n_tokens_neox': 4989, 'n_words': 1355, 'path': 'Coursera_Advanced_Machine_Learning/1_Introduction_to_Deep_Learning/W3_CNN/cnn_with_cifa10.ipynb', 'repo_name': 'dangkhoa1992/Advanced-Machine-Learning-Specialization', 'size': '15361'}